Qwen 27B FP8 Quantized Model Runs on 48GB GPU with vLLM 0.20.1
Run Qwen3.6‑27B‑FP8 on a 48‑GB GPU using vLLM 0.20.1 and CUDA 12.9 to achieve ~80 TPS with BF16 KV cache and Blackwell FP8 acceleration.
Set up a 48‑GB GPU environment, install vLLM 0.20.1, CUDA 12.9, download Qwen3.6‑27B‑FP8, configure BF16 KV cache, enable Blackwell FP8 acceleration, and benchmark for 60‑90 TPS.
Summary
A new FP8‑quantized variant of Qwen3.6‑27B has been released, enabling the 27‑billion‑parameter model to run on a single 48‑GB GPU with Blackwell‑accelerated FP8 and non‑quantized BF16 KV cache.
The setup uses vLLM 0.20.1, CUDA 12.9, and a 200k‑token BF16 KV cache that delivers 1.09× concurrency, while the MTP=2 configuration yields 60‑90 tokens per second for code‑generation workloads.
Key configuration flags include VLLM_USE_FLASHINFER_MOE_FP8=1, VLLM_TEST_FORCE_FP8_MARLIN=1, and enabling auto‑tool‑choice with qwen3_coder parser. The deployment also employs flashinfer attention backend, async scheduling, and a full‑piecewise cuDAG capture mode for compilation.
Benchmarks show that the FP8 variant maintains long‑form Claude‑style sessions without early compaction or endless loops, thanks to the non‑quantized KV cache. Developers can expect roughly 80 TPS in interactive mode, making the model suitable for daily coding assistance.
Key changes
- vLLM 0.20.1 used with CUDA 12.9
- Qwen3.6‑27B‑FP8 variant with Blackwell FP8 acceleration
- BF16 KV cache configured for 200k tokens, yielding 1.09× concurrency
- MTP=2 configuration delivers 60‑90 TPS in interactive mode
- Flashinfer attention backend and async scheduling enabled
- Full‑piecewise cuDAG capture mode for compilation
- Auto‑tool‑choice enabled with qwen3_coder parser
- Non‑quantized KV cache reduces early compaction errors