Optimizing llama-server on Raspberry Pi with ROCm and draft-mtp
Configure llama‑server with draft‑mtp and ROCm for efficient inference on low‑power GPUs.
Apply the provided llama‑server configuration to your Pi setup for optimized inference.
Summary
A user shares a llama‑server configuration that runs on a Raspberry Pi with ROCm devices, enabling flash‑attn, fit, and draft‑mtp for efficient inference. The config uses two RX 9070 XTs PCIe 5.0 x8, each power‑limited to 235 W, and sets ctx‑size to 131072 with a temperature of 0.6. The spec‑type is draft‑mtp with spec‑draft‑n‑max 2, and chat‑template‑kwargs preserve‑thinking is true. The user performed a debugging session that showcased the model’s agentic capabilities, including logging, service orchestration, and code analysis. The setup demonstrates that low‑power GPUs can handle large context sizes when properly configured.
The post provides a practical example of how to tune llama‑server for performance on constrained hardware while maintaining agentic behavior.
Key changes
- llama‑server config uses ROCm0, ROCm1 devices
- flash‑attn enabled, fit true, ctx‑size 131072
- spec‑type draft‑mtp with spec‑draft‑n‑max 2
- two RX 9070 XTs PCIe 5.0 x8 power‑limited to 235 W
- agentic debugging demonstrates logging and service orchestration
- sm tensor enabled for efficient memory usage