Qwen3.6‑27B NVFP4 on RTX 5090 Achieves ~70 tok/s at 200k Context with vLLM

A performance benchmark on a single RTX 5090 with 32 GB VRAM runs the Qwen3.6‑27B‑NVFP4 model (Peutlefaire/Qwen3.6‑27B‑NVFP4) using vLLM 0.20.1.dev0+g88d34c640.d20260502, Torch 2.13.0.dev20260430+cu130, and driver 595.58.03. The setup employs compressed‑tensors quantization, flashinfer attention backend, and an fp8_e4m3 KV cache, with MTP enabled for 3 speculative tokens. The author reports a validated 200k context depth, achieving 65–75 tok/s generation and a 70 s TTFT in a 10‑run stability test, with a mean prefill of 2,883 tok/s and mean generation of 73.6 tok/s. Prefix caching reduces TTFT from 68.8 s (cold) to 2.8 s (warm) at 200k context, though prefill throughput drops from 2,911 to 761 tok/s. The vLLM log shows a mean MTP acceptance length of 2.28, an average draft acceptance of 42.7 %, and a maximum GPU KV cache usage of 88.0 %. The article lists detailed vLLM arguments, including --speculative-config, --kv-cache-dtype, and --max-model-len 230400, and notes that the model reports max_model_len 230400 but only 200k was validated. The author cautions that NVFP4 global scales may reduce accuracy and that prefix caching with Mamba cache align mode remains experimental.

The benchmark uses llama‑benchy 0.3.7 with a 2048‑token prompt and 480‑token generation, measuring prefill and generation rates across context depths from 0 to 200k. The 200k stability run confirms consistent performance with a standard deviation of 13.5 tok/s for generation. The author invites others running Qwen3.6‑27B on RTX 5090 to share their parameters and MTP settings, especially regarding max_num_batched_tokens.