PipelineRL Migrates vLLM from 0.8.5 to 0.18.1, Fixing Train‑Inference Mismatch

PipelineRL recently migrated its inference engine from vLLM 0.8.5 to the new 0.18.1 release, a change that required extensive backend adjustments to preserve training dynamics. The migration uncovered a train‑inference mismatch caused by four key differences: v1’s default logprobs-mode returned raw logits instead of processed probabilities, the new runtime enabled prefix caching and async scheduling by default, the inflight weight‑update path behaved differently, and the final projection used fp32 logits in v0 but not v1. By setting logprobs-mode=processed_logprobs, disabling prefix caching and async scheduling, aligning the weight‑update sequence with engine.pause_generation(mode="keep", clear_cache=False), and enabling fp32 lm_head, the authors restored parity across clip rate, KL, entropy, and reward curves, bringing the v1 run back in line with the v0 reference. The study also highlighted that similar mismatches can surface in PPO, GRPO, or any online RL system that treats rollout‑side logprobs as part of the optimization target, underscoring the importance of backend correctness before applying objective‑side corrections.