Prompt Processing vs Generation Speed: A Real-World Benchmark
Benchmark your own prompt processing times and adjust caching or model selection to reduce prefill latency.
Benchmark your own prompt processing times and adjust caching or model selection to reduce prefill latency.
Summary
The author notes that most discussions focus on token generation speed, but the real bottleneck is prompt processing. They benchmarked Qwen 27B Q6 on their GPU, achieving ~15 tokens per second generation, which is acceptable, but prefill only ~200 tokens per second. They spend more time waiting for the model to ingest the prompt than generating output, even with prompt caching. They work on agentic tasks where the model must ingest a large codebase before acting, so prefill latency is critical. Normal chat has small context, so generation speed matters less.
The post highlights that prompt processing can dominate total wall‑clock time for typical use cases, especially when large contexts are required. It invites others to share their own experiences and suggests re‑examining caching strategies or model selection to reduce prefill latency.
Key changes
- Generation speed ~15 tokens/s is acceptable but prefill only ~200 tokens/s.
- Qwen 27B Q6 on current GPU yields ~15 t/s generation, ~200 t/s prefill.
- Prompt caching improves but prefill remains bottleneck.
- Agentic tasks require large context, making prefill latency critical.
- Normal chat contexts are small, so generation speed matters less.