llama.cpp B9274 Fixes MTP VRAM Leak
Patch llama.cpp to B9274 to free draft resources and prevent VRAM leaks in MTP models.
Patch to B9274 and test MTP workloads for VRAM stability.
Summary
The latest llama.cpp release B9274 addresses a long‑standing VRAM leak that occurs when using Multi‑Token Prediction (MTP) models. The leak was caused by the server_context_impl::destroy() routine only cleaning up the main model and context, leaving the speculative decoder (spec), draft context (ctx_dft), and draft model (model_dft) allocated on the GPU. Each sleep/resume cycle would allocate new GPU buffers without freeing the old ones, eventually exhausting memory and crashing the server. The patch now explicitly resets spec, ctx_dft, and model_dft before calling llama_init.reset(), ensuring a clean shutdown and preventing use‑after‑free errors.
This change eliminates the out‑of‑memory crashes that were common in production deployments of MTP workloads. It also improves stability for developers who rely on the draft‑model feature for faster inference. The update is a minor patch but critical for any production server running MTP models. Users should upgrade to B9274 immediately to avoid memory exhaustion.
Key changes
- Fixes VRAM leak in MTP models by freeing speculative decoder, draft context, and draft model during destroy()
- Adds explicit reset of spec, ctx_dft, and model_dft before llama_init.reset()
- Prevents use‑after‑free by ensuring proper cleanup order
- Addresses issue where sleep/resume cycles allocate new GPU resources without freeing old ones
- Reduces out‑of‑memory crashes in servers using MTP