MTP Boosts Gemma 4 and Qwen 3.6 Inference Speeds by Up to 3×

Multi‑Token Prediction (MTP) has become the latest breakthrough for accelerating large language models, with several independent teams demonstrating significant speed gains across a range of architectures. Google’s release of MTP drafters for the Gemma 4 family claims up to a three‑fold increase in token throughput without sacrificing quality. The approach pairs a heavy target model, such as Gemma 4 31B, with a lightweight drafter that predicts several future tokens before the target verifies them. By sharing the KV cache and activations, the drafter decouples generation from verification, reducing memory‑bandwidth bottlenecks on GPUs ranging from NVIDIA RTX PRO 6000 to Apple Silicon and A100. The drafters are distributed under an Apache 2.0 license and can be integrated with MLX, vLLM, SGLang, and Ollama, making the technology accessible to developers building coding assistants, autonomous agents, and mobile applications.

A separate engineering effort has grafted MTP onto the Qwen 3.6 27B model using Unsloth’s UD XL quantizations. The base model remains in low‑bit quantization, while three draft heads stay at Q8 to preserve speculative accuracy. The result is a 2.5‑times increase in token throughput compared to the same UD XL GGUF without MTP. The work leverages speculative decoding support from llama.cpp PR #22673, merges it onto the master branch, and compiles a custom llama‑server that accepts the --spec‑type mtp flag. The MTP layers add negligible VRAM overhead, and the draft tokens are largely accepted, confirming the efficiency of speculative decoding in this context. The author has released a convert.py script, build instructions, and the grafted GGUF files, enabling replication on other models.

The AtomicBot team has added MTP to the LLaMA.cpp inference engine and released quantized Gemma 4 assistant models in GGUF format. On a MacBook Pro M5Max, the Gemma 26B model with MTP achieved 138 tokens/s versus 97 tokens/s without the feature, a 40 % speedup. The improvement is demonstrated with a simple Python recursion prompt for the nth Fibonacci number. The patch is available on GitHub (https://github.com/AtomicBot‑ai/atomic‑llama‑cpp‑turboquant) and the quantized models can be downloaded from Hugging Face under the AtomicChat collection.

Collectively, these developments show that MTP can be applied across multiple frameworks—Gemma, Qwen, and LLaMA.cpp—using open‑source tooling such as llama.cpp and GGUF. The speed gains, ranging from 40 % to 3×, are achieved with minimal additional memory usage, making the technology attractive for both high‑performance servers and edge devices. As more developers adopt MTP, it is likely to become a standard component of next‑generation inference pipelines.