Speculative Decode vs Ngram in llama.cpp: Can They Be Combined?
Verify that passing both speculative decode and ngram flags results in only ngram being active; if you need both, modify the code or wait for a future patch.
Verify that passing both flags results in only ngram being active; if you need both, modify the code or wait for a future patch.
Summary
Speculative decode with Qwen 3.6 27B works well in llama.cpp, but the author notes that when both speculative decode and ngram flags are passed, only ngram is active. The ngram approach speculates tokens quickly, which improves agentic coding by avoiding repeated verbatim code. The limitation appears to be an implementation detail in the command‑line parser rather than a fundamental restriction of the models. The author wonders whether the two techniques can be combined simultaneously or if a future patch will enable it. No fix has been announced yet, and the issue is currently unresolved. The post invites discussion on whether the limitation is due to a bug or a design choice. Developers using llama.cpp should test the flag combination to confirm the current behaviour. If simultaneous use is required, they may need to modify the code or wait for a future update.
Key changes
- Speculative decode works with Qwen 3.6 27B in llama.cpp
- When both speculative decode and ngram flags are passed, only ngram is active
- The limitation is likely an implementation detail in command‑line parsing
- No fix has been announced; issue remains unresolved
- The author suggests combining the two techniques may be possible with a future patch
- Developers should test flag combinations to confirm current behaviour