Quantizing MTP KV Cache for Qwen3.7-27B-Q8_0 Provides Free Lunch
Quantize the MTP KV cache on Qwen3.7‑27B‑Q8_0 with –cache-type-k-draft q8_0 –cache-type-v-draft q8_0 and benchmark using –spec-type draft-mtp –spec-draft-n-max 3 to confirm VRAM savings and performance gains.
Quantize the MTP KV cache on your Qwen3.7‑27B‑Q8_0 deployment using –cache-type-k-draft q8_0 –cache-type-v-draft q8_0 and benchmark with –spec-type draft-mtp –spec-draft-n-max 3 to confirm VRAM savings and performance.
Summary
The MTP layer in Qwen3.6/3.5 models consumes extra VRAM, but its KV cache can be quantized to reduce memory usage. By applying –cache-type-k-draft q8_0 and –cache-type-v-draft q8_0, the cache shrinks while still supporting the same draft‑MTP workflow. Benchmarks on Qwen3.7‑27B‑Q8_0 with –spec-type draft-mtp –spec-draft-n-max 3 show an aggregate accept rate of 0.735 to 0.7411 and wall‑time reductions from 49.46s to 38.29s when using tensor parallelism.
The tests were run on two Mi50 GPUs, each with 32GB PCIe 4.0 x8, and compared three configurations: baseline, baseline with quantized KV cache, and tensor parallelism with quantized KV cache. The quantized KV cache consistently lowered VRAM requirements and improved throughput without sacrificing draft acceptance.
These findings suggest that quantizing the MTP KV cache is a practical way to fit more context into the same hardware, especially when deploying large Qwen models in resource‑constrained environments.
Key changes
- MTP layer requires extra VRAM but its KV cache can be quantized
- Using –cache-type-k-draft q8_0 and –cache-type-v-draft q8_0 reduces memory footprint
- Benchmark shows aggregate accept rate 0.735–0.7411 with quantized cache
- Wall‑time drops from 49.46s to 38.29s when using tensor parallelism
- Spec-type draft-mtp with spec-draft-n-max 3 is the tested configuration
- Tests performed on 2xMi50 32GB PCIe 4.0 x8 GPUs
- Tensor parallelism (sm tenor or sm tensor) yields similar performance
- Quantized KV cache allows more context on the same hardware