Qwen 3.6 35B on AMD 5800X Achieves 1000+ TPS but Lacks Context Compaction

The post describes running Qwen 3.6 35B on an AMD 5800X with 96 GB DDR4 and an RX 6800XT 16 GB under Ubuntu 26.04 compiled with ROCM 7.2.2, achieving over 1,000 tps for prompt processing and 15–22 tps for token generation. The command line uses the ngram‑mod speculative decoding mode with n‑match 24, n‑min 12, n‑max 48, and fit‑on with a 230,000‑token context and a 256‑token target. Despite the impressive throughput, the author reports that all coding agents—including kilo, opencode, and pi—fail to perform context compaction without invalidating the entire cache, causing a full re‑processing of the prompt. This limitation becomes a bottleneck when the context approaches the 200‑k token range, even though the GPU can still process prompts at 1,000 tps. The issue is not yet resolved, and the author asks the community for solutions or workarounds. The post highlights that the problem is specific to local AI setups using ROCM and not a general LLM limitation. The author also notes that the setup is suitable for professional use, as the model runs efficiently on the described hardware.

The article ends with a call for a fix, suggesting that agents need to support true context compaction or that users disable the fit option to avoid cache invalidation. It also implies that the current state of local AI tooling may need improvement to handle large contexts without sacrificing performance. The author also provides a detailed command and hardware specs as a reference point for developers looking to replicate or troubleshoot similar setups.