Prompt Caching for RL Training Achieves 7.5× Speedup on Long‑Prompt Workloads
Implement prompt caching in your RL training pipeline to compute the prompt once and reuse it for all G responses, reducing token processing by up to 7.5×.
Add prompt caching logic to your RL training code and test on Qwen3.5‑4B to verify the claimed speedups.
Summary
Most open‑source RL engines naively repeat the prompt and response for every sample in a group, which is wasteful for long‑prompt, short‑completion tasks. With a 1,000‑token prompt and 100‑token response at G=8, the engine processes 8,800 tokens while only 1,800 are unique, wasting roughly five times the compute. The proposed fix computes the prompt once and then generates all G responses, analogous to inference prefix caching but with gradients flowing back through the prompt. This requires careful handling of causal attention, especially for full versus linear attention layers.
The technique was evaluated on Qwen3.5‑4B, yielding speedups of 7.5× for 16k prompt / 64 output, 7.3× for 16k / 128, 5.4× for 16k / 1k, and 1.7× for 8k / 4k. These results demonstrate that prompt caching can dramatically reduce token processing for RL training on long‑prompt workloads.
Implementing prompt caching can bring significant efficiency gains, but developers must ensure that gradient flow and attention mechanics remain correct across different attention implementations.
Key changes
- RL engines currently repeat prompt+response for each sample
- 1000‑token prompts + 100‑token responses at G=8 process 8,800 tokens vs 1,800 unique
- Prompt caching computes the prompt once, then G responses
- Requires gradient flow through the prompt, breaking causal attention
- Works with full and linear attention layers
- Qwen3.5‑4B speedups: 16k/64 → 7.5×, 16k/128 → 7.3×, 16k/1k → 5.4×, 8k/4k → 1.7×