Increasing Micro‑Batch Size in llama.cpp Boosts Prompt Processing for gpt‑oss‑120b‑F16.gguf on RTX 3090
Increase llama.cpp micro‑batch size (-ub) up to 8192 and set --n‑cpu‑moe to 28 on RTX 3090 to boost prompt processing from ~380 to ~2091 tok/s, accepting a ~7 % drop in generation speed.
Increase llama.cpp micro‑batch size to 8192 and set --n‑cpu‑moe to 28 on RTX 3090 to maximize prompt throughput, monitoring generation speed.
Summary
A user tuned llama.cpp on a 24 GB RTX 3090 to run the gpt‑oss‑120b‑F16.gguf model. By increasing the physical micro‑batch size (-ub) from the default 512 up to 8192 and raising --n‑cpu‑moe accordingly (from 26 to 28), prompt throughput rose from 380 tok/s to 2091 tok/s—a 5.5× gain. Generation speed fell modestly from 32.3 tok/s to 30.1 tok/s, a 7 % reduction. Larger ubatches required more GPU compute workspace, so the trade‑off favors prompt‑heavy workloads.
The author notes that the results are informal tuning rather than a controlled benchmark, and that the 8192 ubatch run used a pp8192 prefill, so comparisons should be treated cautiously.
These findings suggest that for inference workloads dominated by prompt processing, significantly larger micro‑batches can dramatically improve throughput at the cost of a small generation slowdown.
Key changes
- Raising -ub from 512 to 8192 increases prompt throughput from 380 to 2091 tok/s (5.5×)
- Corresponding --n‑cpu‑moe must increase from 26 to 28 to fit GPU memory
- Generation speed drops from 32.3 to 30.1 tok/s (~7 %)
- Larger ubatch requires more GPU compute workspace
- Trade‑off: prompt‑heavy workloads benefit, generation slower
- Default llama.cpp settings: -b 2048, -ub 512, --n‑cpu‑moe 26
- Results are informal tuning, not a controlled benchmark