Multi-Token Prediction (MTP) for LLaMA.cpp – Gemma 4 Speedup by 40%
Patch your LLaMA.cpp repository with the AtomicBot‑ai/atomic‑llama‑cpp‑turboquant fork and enable Multi‑Token Prediction for Gemma 4 models to gain ~40% speedup.
Patch your LLaMA.cpp repository with the AtomicBot‑ai/atomic‑llama‑cpp‑turboquant fork and enable Multi‑Token Prediction for Gemma 4 models to achieve ~40% speedup.
Summary
A new Multi-Token Prediction feature has been added to LLaMA.cpp, allowing the model to draft multiple tokens in a single pass. The update includes quantized Gemma 4 assistant models converted to GGUF format, which can be downloaded from the AtomicChat Hugging Face collection. On a MacBook Pro M5Max, the Gemma 26B model with MTP achieved 138 tokens per second, a 40% increase over the baseline 97 tokens per second.
The patch is available in the AtomicBot‑ai/atomic‑llama‑cpp‑turboquant GitHub repository, and a local AI application, atomic.chat, demonstrates the new capability. Developers can now integrate the updated LLaMA.cpp build to benefit from faster inference on Gemma 4 models.
Key changes
- Multi‑Token Prediction added to LLaMA.cpp
- Gemma 4 assistant models quantized to GGUF format
- Gemma 26B with MTP drafts tokens 40% faster (138 vs 97 t/s)
- Performance tested on MacBook Pro M5Max
- GGUF models available at https://huggingface.co/collections/AtomicChat/gemma-4-assistant-gguf
- Local AI app atomic.chat demonstrates the feature
- Patched llama.cpp repo at https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant