High GPU Utilization Doesn’t Mean High Throughput: vLLM 0.18.0 on RTX 4090
Check your GPU utilization metrics: nvidia‑smi duty‑cycle counts kernels, not useful work; correlate with kernel runtime, off‑CPU time, and NCCL waits to diagnose throughput drops.
Patch your monitoring stack to include cause‑side metrics such as cudaLaunchKernel latency, sched_switch counts, and off‑CPU time; adjust vLLM request parameters to avoid large logprobs and n_completions; test with smaller batch sizes to confirm throughput.
Summary
A vLLM 0.18.0 server running the Qwen2.5‑0.5B‑Instruct model on a TensorDock RTX 4090 reported 97 % GPU utilization for an 8‑minute window while token throughput fell from 2,180 tok/s to 730 tok/s and then recovered. The root cause was a request with “n_completions=8” and “logprobs=20” that expanded each decode step into 8 sequences with a full‑vocabulary softmax (~150 K tokens), blocking the dispatcher thread for 9‑11 s at a time. This demonstrates that nvidia‑smi’s GPU‑Util counter is a duty‑cycle metric that only shows that at least one kernel ran, not that the GPU was doing useful work. The article identifies five common failure modes that can produce high utilization but low throughput: prefill/decode imbalance, collective‑communication waits, I/O stalls, CPU contention on the engine thread, and memory‑bandwidth saturation. By correlating CUDA runtime calls, driver API calls, Linux scheduler tracepoints, and off‑CPU time, the author shows that 18 % of cudaLaunchKernel calls were stalled by off‑CPU events, causing the observed throughput drop.
The analysis recommends moving beyond single‑point utilization metrics and adopting cause‑side signals such as kernel‑runtime distributions, cudaLaunchKernel p99 spread, sched_switch counts, and off‑CPU time per dispatcher PID. It also highlights the value of DCGM counters (SM_ACTIVE, TENSOR_ACTIVE, MEM_COPY_UTIL) for finer granularity, while noting that they still cannot reveal whether the GPU is idle due to dispatch delays. For developers, the key takeaway is to instrument both the GPU and host layers to diagnose performance regressions accurately.
Key changes
- nvidia‑smi GPU‑Util is a duty‑cycle counter that can read 97 % while throughput drops
- vLLM 0.18.0 with Qwen2.5‑0.5B‑Instruct on RTX 4090 shows 97 % utilization but 3× token throughput drop
- Five failure modes identified: prefill/decode imbalance, collective‑communication wait, I/O stall, CPU contention, memory‑bandwidth saturation
- 18 % of cudaLaunchKernel calls stalled due to off‑CPU events on the dispatcher thread
- DCGM counters (SM_ACTIVE, TENSOR_ACTIVE, MEM_COPY_UTIL) provide finer granularity but still lack dispatch insight
- Cause‑side metrics such as cudaLaunchKernel p99 spread, sched_switch counts, and off‑CPU time are essential for diagnosis
- Prefill/decode imbalance can stall short‑context requests even with high GPU utilization
- Collective‑communication wait in distributed training can show high utilization on fast ranks while throughput is bounded by the slowest rank