DFlash Speculative Decoding Benchmark on vLLM 0.19.2rc1 with Gemma 4‑26B
Configure vLLM with num_speculative_tokens=13 and max_num_batched_tokens=8192 to achieve ~2.56× speedup on RTX 5090 for Gemma 4‑26B.
Set num_speculative_tokens=13 and max_num_batched_tokens=8192 in vLLM config, run the benchmark, and monitor latency tail.
Summary
A benchmark was run on vLLM 0.19.2rc1 with an RTX 5090 32 GB GPU, using the Gemma 4‑26B‑A4B‑it‑AWQ‑4bit as the main model and the Gemma 4‑26B‑A4B‑it‑DFlash as the draft. Baseline throughput was ~228 output tok/s with ~4 455 ms latency. The best practical setting used num_speculative_tokens = 13 and max_num_batched_tokens = 8192, achieving ~578 tok/s and ~1 738 ms latency, a 2.56× speedup. The optimal latency setting did not always give the best tail; 8192 tokens yielded a cleaner tail than 4096. The author shares a video, script, and recommended command for others to replicate the results.
Key changes
- GPU: RTX 5090 32 GB
- vLLM version 0.19.2rc1
- Main model: gemma‑4‑26B‑A4B‑it‑AWQ‑4bit
- Draft model: gemma‑4‑26B‑A4B‑it‑DFlash
- Baseline output tok/s ~228, latency ~4 455 ms
- Best setting: num_speculative_tokens = 13, max_num_batched_tokens = 8192
- Output tok/s ~578, latency ~1 738 ms, speedup 2.56×
- 8192 tokens gives cleaner tail than 4096