Benchmarking vLLM, SGLang, and llama.cpp on a 7‑GPU Cluster Shows vLLM Dominates Mixed‑GPU Prefill
Test vLLM on mixed GPU clusters for long context prefill, and use VLLM_PP_LAYER_PARTITION to balance uneven splits.
Deploy vLLM on your mixed GPU clusters and set VLLM_PP_LAYER_PARTITION to balance compute load for long‑context prefill.
Summary
A heterogeneous 7‑GPU cluster consisting of one RTX PRO 6000 96 GB, one PRO 5000 48 GB, two 5090 32 GB, and three modded 4090 48 GB was used to benchmark inference engines. The tests employed 4‑bit weights, with NVFP4 for vLLM and SGLang and MXFP4 for llama.cpp. vLLM significantly outperformed the others on mixed multi‑GPU setups for long‑context prefill, while llama.cpp lagged 4‑6× behind due to pipeline bubbles. SGLang performed well on pure Blackwell setups but crashed when an Ada card was added because it lacks an FP4 fallback.
vLLM handled uneven GPU splits by using the VLLM_PP_LAYER_PARTITION environment variable, balancing compute load between fast Blackwells and slower 4090s that emulate FP4, which eliminated bottlenecks and yielded massive speedups even on a 397B model. The benchmark table shows vLLM achieving 18.9 s TTFT and 6,212 t/s on a 6‑GPU mixed setup, compared to 77.0 s and 1,065 t/s for llama.cpp. SGLang crashed on mixed setups but matched vLLM on pure Blackwell configurations. The results suggest that vLLM chunked prefill and manual layer partitioning are essential for mixed‑GPU clusters relying on pipeline parallelism.
Key changes
- vLLM outperforms others on mixed multi‑GPU long‑context prefill
- llama.cpp suffers 4‑6× slowdown due to pipeline bubbles
- SGLang crashes on mixed setups lacking FP4 fallback
- vLLM balances uneven splits via VLLM_PP_LAYER_PARTITION, eliminating bottlenecks
- vLLM achieves 18.9 s TTFT and 6,212 t/s on 6‑GPU mixed cluster
- SGLang matches vLLM on pure Blackwell setups
- vLLM handles 397B model with 9.8 s TTFT and 7,683 t/s on uneven split
- Manual layer partitioning is crucial for mixed‑GPU performance