NVLink Boosts vLLM Throughput on 4× RTX 3090 – TP=2 Wins Over TP=4
Configure vLLM to pin tensor parallelism to NVLink pairs (TP=2 on 0+2) for ~50% throughput gains at high concurrency.
Configure vLLM to pin TP=2 to NVLink pairs; avoid TP=4 on 3090 chassis; run separate services per NVLink pair.
Summary
A benchmark on four RTX 3090 GPUs bonded with NVLink between pairs 0↔2 and 1↔3 shows that pinning tensor parallelism (TP) to the NVLink pair yields a 25% throughput increase at single-stream concurrency and a 53% increase at concurrency four, compared to running TP=2 over PCIe. Using TP=4 across all GPUs actually reduces throughput by 13% at single-stream and 30% at high concurrency, due to the all-reduce ring being bottlenecked by slow PCIe links. The study employed vLLM 0.20.1, transformers 5.7.0, CUDA 12.8, and the 27B Qwen3.6-27B-AWQ-BF16-INT4 model, which includes a speculative decoding head. Multi-Token Prediction (MTP) speculative decoding maintained a 70–79% acceptance rate across all topologies, indicating the bottleneck was communication, not draft quality. The results suggest that for two-pair NVLink 3090 boxes, the optimal serving pattern is two TP=2 services, one per NVLink pair, rather than a single TP=4 service.
These findings are valuable for teams deploying large language models on multi-GPU setups, as they demonstrate the importance of topology-aware configuration for maximizing throughput.
Key changes
- 4× RTX 3090 with NVLink between 0↔2 and 1↔3.
- TP=2 on NVLink pair yields +25% throughput at concurrency 1 and +53% at concurrency 4.
- TP=4 across all GPUs is slower than TP=2 NVLink (−13% at conc1, −30% at conc4).
- MTP speculative decoding acceptance rate remains 70–79% across topologies.
- vLLM 0.20.1, transformers 5.7.0, CUDA 12.8 used.
- Model cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 (27B, hybrid attention, MTP head).
- NVLink advantage reduces TTFT by 45% at conc4 compared to PCIe.
- All-reduce ring bottleneck explains TP=4 slowdown.