Paper advocates W4A4 quantization for prefilling, not decoding
Use W4A4 quantization for prefill to gain 4x speed, avoid for decoding due to error accumulation.
Implement W4A4 quantization for context encoding and keep decoding at high precision.
Summary
A recent paper discusses the limitations of NVFP4 decoding speed and proposes parallel decoding as a more effective optimization. The authors argue that prefilling, which processes a fixed input sequence in parallel, is well suited to aggressive quantization because errors do not accumulate across tokens. In contrast, decoding is highly error‑sensitive, and applying aggressive W4A4 quantization to the full autoregressive process leads to brittle token choices. The paper introduces Mix‑Quant, which quantizes only the context encoding while keeping decoding at high precision. The authors also note that NVFP4 decoding does not achieve the desired memory bandwidth utilization, and that parallel decoding offers a more promising path.
The findings suggest that W4A4 should be used for prefill to achieve a theoretical 4× speedup, but decoding should remain in high precision to avoid error accumulation.
Key changes
- W4A4 suitable for aggressive prefill
- Avoid W4A4 for decoding to prevent error accumulation
- Mix‑Quant quantizes only context encoding
- NVFP4 decoding speed limited
- Parallel decoding preferred
- Quantization redundancy differs between prefill and decoding