DeepSeek-V4-Flash MTP Self-Speculation Boosts Decode Speed on RTX 6000 Max-Q
Patch vLLM to restore MTP head in DeepSeek-V4-Flash quant; run on 2x RTX 6000 Max-Q with --disable-custom-all-reduce and NCCL tuning for optimal speed.
Apply patched vLLM fork, run DeepSeek-V4-Flash quant with --disable-custom-all-reduce and NCCL tuning; monitor TTFT and TPS.
Summary
DeepSeek-V4-Flash W4A16-FP8 quant was released with its MTP head stripped during load, but a patched vLLM restores the head and unlocks speculative decoding. Decoding throughput jumps from 52.85 tok/s (no MTP) to 85.52 tok/s at 524k 2-stream and ~111 tok/s at 128k single-stream, a 62 % and 110 % increase respectively. The model totals 671 B parameters with 32 B active, fitting comfortably on two 96 GB RTX 6000 Max-Q GPUs. Quantization involved GPTQ of 768 routed-expert tensors, 5 attention projections, and shared experts in BF16/FP32, while Max-Q specific fixes require --disable-custom-all-reduce to avoid deadlock. NCCL tuning (LL protocol, Ring algorithm, 8 channels, 512 threads) reduces TTFT from ~155 ms to ~91 ms. Limitations include TP = 2 only, a single speculative token, and a reasoning parser caveat that splits content and reasoning_content. The patched vLLM fork and detailed run command are provided for reproducibility.
Key changes
- MTP head stripped during load, patched vLLM restores it
- Decoding throughput increases from 52.85 to 85.52 tok/s (524k 2-stream) and ~111 tok/s (128k single-stream)
- Model totals 671 B parameters, 32 B active, fits on 2x 96 GB RTX 6000 Max-Q
- Quantization: GPTQ of 768 routed-expert tensors, 5 attention projections, shared experts BF16/FP32
- Max-Q fix: use --disable-custom-all-reduce to avoid deadlock
- NCCL tuning reduces TTFT from ~155 ms to ~91 ms
- Limitations: TP=2 only, single speculative token, reasoning parser caveat
- Patched vLLM fork and run command provided