MTP performance drop in llama.cpp merged, ik_llama.cpp restores speed
Switch to ik_llama.cpp with CPU offloading and adjust launch parameters to regain MTP performance.
Migrate production llama.cpp instances to ik_llama.cpp and apply the provided launch flags to restore MTP throughput.
Summary
A user experienced a performance drop after the MTP PR was merged into llama.cpp while using an RTX 4070 Super 12GB GPU. The merged PR caused MTP performance to fall to barely above non‑MTP levels. Switching to ik_llama.cpp, which supports CPU offloading and is better optimized for MTP, resulted in a 23% speed increase, raising the average tok/s from 89.76 to 110.24. The user provided detailed benchmark results for both llama.cpp and ik_llama.cpp, showing consistent improvements across multiple request types. Launch parameters for ik_llama.cpp include --fit-margin 1664, --ctx-size 131072, and various cache‑type settings. The user also noted that increasing --fit-margin to 1792 or 2048 can mitigate out‑of‑memory errors.
The post demonstrates that the new ik_llama.cpp implementation can recover MTP performance on 12GB GPUs when configured correctly.
Key changes
- MTP PR merged caused performance drop
- Switch to ik_llama.cpp restores 23% speed boost
- Average tok/s increased from 89.76 to 110.24
- Launch flags include --fit-margin 1664, --ctx-size 131072, --cache-type-k q8_0
- CPU offloading improves throughput
- Fit margin adjustment mitigates OOM errors