MTP and Tensor Parallelism Boost Qwen3.6-27B Performance on ROCm 7.2

The author reports a 1.5× speedup using MTP grafting on Qwen3.6‑27B‑MTP‑Q4_1.gguf, and up to 2× when combined with tensor parallelism on ROCm 7.2. The setup uses CachyOS (Arch Linux), the llama.cpp fork https://github.com/skyne98/llama.cpp‑gfx906, and the PR https://github.com/ggml-org/llama.cpp/pull/22673. The command line includes --spec-type mtp --spec-draft-n-max 2, --fit on –fitt 256, and runs on ROCm0, ROCm1 with no‑mmap and no‑warmup. The benchmark script shows token‑per‑second rates of 26.2 tok/s for stock, rising to 39.6 tok/s with MTP alone and 59.8 tok/s when MTP and tensor parallelism are combined. Aggregate accept rates climb from 0.7806 stock to 0.799 combined, and wall‑time drops from 51.42 s to 32.24 s over nine requests.

Real‑world testing on an 18k‑token coding prompt reveals that the 60 t/s figure is only observable for very short prompts; the combined MTP+tensor run achieves 205 s total time versus 390 s for stock, a 2× speedup overall. However, the author notes a regression in prefill speed that is currently being addressed. The benchmark data also shows a 4.40 ms per token prefill time with MTP versus 2.18 ms with tensor parallelism alone. The combined run reduces prefill to 2.59 ms per token while maintaining high acceptance rates. The author also highlights that the MTP grafting was achieved by transplanting the technique onto Bartowski's quant model, as described in a Reddit post.

The author encourages readers to try the provided fork and command on their own hardware, noting that the speed gains are most pronounced on older GPUs lacking compute resources. The 1.5× baseline improvement and the 2× combined improvement demonstrate the potential of MTP and tensor parallelism for large‑model inference on ROCm. The detailed token‑rate tables and aggregate statistics provide a clear benchmark for comparison.