Google Unveils Multi‑Token Prediction Drafters for Gemma 4, Delivering Up to 3× Faster Inference

Google has introduced Multi‑Token Prediction (MTP) drafters for its Gemma 4 family of language models, a development that promises to slash inference times while preserving the models’ reasoning quality. The drafters pair a lightweight “drafter” with either the 26‑billion‑parameter Mixture‑of‑Experts (MoE) or the 31‑billion‑parameter dense target models. By speculatively generating several tokens in advance and sharing activations and key‑value caches, the system eliminates redundant context recomputation. The target model then verifies all suggested tokens in parallel, ensuring that the speed gains do not compromise output fidelity.

Benchmarks across a range of inference frameworks—including LiteRT‑LM, MLX, Hugging Face Transformers, and vLLM—show that the MTP drafters can achieve up to a three‑fold increase in tokens‑per‑second. On edge‑optimized models such as E2B and E4B, an additional clustering technique in the embedder further accelerates logit calculation. When run on Apple Silicon or Nvidia A100 GPUs with larger batch sizes, users can see an extra 2.2‑fold local speedup. A separate update to the open‑source LLaMA.cpp engine, which now supports MTP, demonstrated a 40 % throughput increase on a MacBook Pro M5Max, raising performance from 97 to 138 tokens per second.

The MTP drafters are released under an Apache 2.0 license and are available on Hugging Face, Kaggle, and the Google AI Edge Gallery for Android and iOS. They are compatible with popular inference engines such as vLLM, MLX, SGLang, and Ollama, making integration straightforward for developers. The verification step guarantees that the accelerated inference remains on par with frontier‑class reasoning and accuracy, allowing teams to deploy faster local AI workloads without sacrificing quality.