Local LLM Prompt Processing Bottlenecks
Review prompt processing speeds on your local LLM to identify bottlenecks.
Test prompt processing speeds on your local LLM setup to confirm bottlenecks.
Summary
Prompt processing is often the real bottleneck in local LLM inference, not decoding. The author notes that a 15k-token prompt takes about 15k tokens to start, yielding roughly 10 tokens per second on a mac mini. Using Qwen3.6 27b, prompt processing of a 64k prompt took over 10 minutes, prompting a switch to the 35b a3b variant. The author wonders whether MTP or other techniques can speed up prompt handling, and questions whether discrete GPU settings alter the bottleneck. The discussion highlights that decoding speed can exceed human eye tracking, but prompt latency remains a limiting factor. No specific fixes are offered, only a call for community insight. The post invites readers to share their experiences with prompt processing on local hardware.
Key changes
- Prompt processing speed ~10 tokens/s for 15k-token prompts on mac mini.
- Qwen3.6 27b prompt processing >10 minutes for 64k prompt.
- Switched to 35b a3b variant for faster prompt handling.
- Bottleneck identified in prompt processing rather than decoding.
- MTP or other methods may improve prompt speed.
- Discrete GPU settings may affect bottleneck.