FastDMS: 5‑8x KV Memory Reduction and 1.5‑2x Speed Boost for LLMs
Implement FastDMS in your inference pipeline to cut KV memory usage by up to 8× and boost decoding speed by 1.5‑2×.
Integrate FastDMS into your LLM inference stack to reduce memory footprint and improve throughput.
Summary
Dynamic Memory Sparsification (DMS) was introduced by NVIDIA, University of Warsaw, and University of Edinburgh researchers, achieving up to 8× KV‑cache compression through learned per‑head token eviction. A reference implementation was built and trained on WikiText‑2 with Llama 3.2 1B, showing a 0.28% PPL improvement and 6.4× compression. The author released FastDMS, an MIT‑licensed implementation that physically reclaims evicted KV slots and is tested on both Qwen 3 8B and Llama 3.2 1B checkpoints. FastDMS uses 5‑8× less KV memory than vLLM BF16 KV at 8K context and decodes 1.5‑2× faster than vLLM BF16. The compression is lossless, with KLD values as low as 0.026 nats/token, and FastDMS outperforms TurboQuant 4‑bit in both speed and memory usage.
FastDMS supports FP8 and BF16 KV types, achieving compression ratios up to 8×. Training the DMS predictors takes roughly 20 minutes on a PRO 6000 GPU. The repository includes the original HF reference, a trainer, and the FastDMS implementation, all available on GitHub.
Key changes
- FastDMS uses 5‑8× less KV memory than vLLM BF16 KV at 8K context
- FastDMS decoding speed 1.5‑2× faster than vLLM BF16
- FastDMS physically reclaims evicted KV slots with compact storage
- DMS eviction applied before FP8 quantization, preserving quality
- FastDMS supports FP8 and BF16 KV types with compression ratios up to 8×
- FastDMS outperforms TurboQuant 4‑bit in speed and memory usage
- Training DMS predictors takes ~20 minutes on PRO 6000 GPU
- FastDMS is MIT‑licensed and available on GitHub