Blackwell LLM Toolkit NVFP4 Configs, Wheels, and Benchmarks for RTX Pro 6000
Clone the Blackwell LLM Toolkit, rebuild the LMCache wheel for Blackwell, and benchmark NVFP4 quantized models like Nemotron‑3‑Nano‑Omni V3 on RTX Pro 6000 to achieve up to 270 tok/s.
Clone https://github.com/elsung/blackwell‑llm‑toolkit, apply the provided configs, rebuild the LMCache wheel, and run bench_harness.py to validate performance on your Blackwell GPU.
Summary
A new GitHub repo, blackwell‑llm‑toolkit, provides NVFP4 quantized model configs, wheels, and benchmarks for Nvidia Blackwell GPUs such as the RTX Pro 6000. The repo includes TRT‑LLM launch flags in configs/trtllm/nemotron‑omni‑v3‑sm120.yaml to enable newer Mamba‑hybrid models, and a rebuilt LMCache wheel that offloads context to SSD, fixing missing sm_120 cubins on Blackwell. Benchmarks show Nemotron‑3‑Nano‑Omni V3 (multimodal) achieving 270 tok/s at 8 k context with TRT‑LLM v1.3.0rc13, while Nemotron‑3‑Nano (text only) reaches 249 tok/s. Other models benchmarked include DeepSeek‑V4‑Flash at 31 tok/s (65 k context), MiniMax‑M2.7‑REAP‑172B at 117 tok/s (196 k context), and MiniMax‑M2.7 W4A16 with LMCache at 20‑22 tok/s (154 k context). The repo also contains bench_harness.py and rapid_bench.py for sustained decode, TTFT, prefill, concurrency, and quality evaluation.
These tools enable developers to run high‑performance, low‑precision LLMs on Blackwell GPUs, with significant throughput gains and SSD‑based KV cache offloading.
Key changes
- Repo provides NVFP4 configs and TRT‑LLM launch flags for Blackwell GPUs
- LMCache wheel rebuilt from source to support Blackwell (missing sm_120 cubins)
- Nemotron‑3‑Nano‑Omni V3 achieves 270 tok/s at 8 k context with TRT‑LLM v1.3.0rc13
- Nemotron‑3‑Nano (text only) achieves 249 tok/s at 8 k context
- DeepSeek‑V4‑Flash achieves 31 tok/s at 65 k context
- MiniMax‑M2.7‑REAP‑172B achieves 117 tok/s at 196 k context
- MiniMax‑M2.7 W4A16 with LMCache achieves 20‑22 tok/s at 154 k context
- MiniMax‑M2.7 W4A16 short‑ctx achieves 22‑25 tok/s at 64 k context