vllm Merges Turboquant Fix for Qwen 3.6
Test the new turboquant options for Qwen 3.6 by running vllm with --kv-cache-dtype turboquant_4bit_nc and verify inference.
Patch vllm to the latest commit from PR #39931 and test Qwen 3.6 with turboquant options.
Summary
The vllm project has merged a fix for the turboquant feature that previously caused a "Not Implemented" error when using Qwen 3.6 models. The pull request #39931 addresses Mamba layer incompatibilities, enabling the library to run Qwen 3.6 with 27B parameters without crashing.
Users can now enable turboquant by passing the --kv-cache-dtype flag with values such as turboquant_4bit_nc, turboquant_k8v4, turboquant_k3v4_nc, or turboquant_3bit_nc. The update was tested on the 27B Qwen 3.6 model, confirming stable inference and reduced memory usage. The new options allow fine‑grained control over quantization precision, letting developers choose between 4‑bit and 3‑bit configurations. The PR also cleans up the code path that previously raised the Not Implemented exception. This change is a direct improvement for developers running large Qwen models locally.
Key changes
- Turboquant fix for Qwen 3.6 resolves a 'Not Implemented' error caused by Mamba layers
- PR #39931 merges the patch into the main branch
- The library now supports Qwen 3.6 with 27B parameters
- New --kv-cache-dtype flag values include turboquant_4bit_nc, turboquant_k8v4, turboquant_k3v4_nc, and turboquant_3bit_nc
- Quantization precision can be selected between 4‑bit and 3‑bit options
- The code path that raised the exception has been cleaned up
- The update was tested and confirmed stable inference on the 27B Qwen 3.6 model