Qwen3.5‑397B‑A17B Runs in 14 GB RAM on M1 Ultra via Paged MoE
Configure the MoE paging with K_override=20, cache_gb=8.0, lazy_load=True, and OUTLIER_MMAP_EXPERTS=0 to run Qwen3.5‑397B‑A17B on a 64GB M1 Ultra with ~14 GB peak RAM.
Set K_override=20, cache_gb=8.0, OUTLIER_MMAP_EXPERTS=0, lazy_load=True, and adjust cache size to avoid command‑buffer overflows when running Qwen3.5‑397B‑A17B on a 64GB M1 Ultra.
Summary
The 397B model occupies 209 GB on disk, contains 512 experts, and uses top‑10 routing per token; a naive load fails on a 64 GB M1, so lazy‑paging keeps only K=20 experts resident and loads the rest from SSD on demand. The engine configuration uses K_override=20, cache_gb=8.0, OUTLIER_MMAP_EXPERTS=0, lazy_load=True, and float16 compute via MLX, which is faster than ternary MPS on Apple Silicon. Performance metrics on a 64 GB M1 Ultra show 1.59 tok/s, 7.91 GB cache RSS peak, 14.04 GB total RSS peak, and 5/5 coherent prompts, making MMLU per GB of RAM the key metric. Smaller tiers on the same hardware include 4B Nano (71.7 tok/s), 9B Lite (53.4), 26B‑A4B Quick (14.6), 27B Core (40.7), 35B‑A3B Vision (64.1), and 397B Plus (1.59). The runtime is built with Tauri + Rust + MLX and is available at outlier.host.
Key changes
- 397B model is 209 GB on disk, 512 experts, top‑10 routing per token
- Lazy‑paging keeps only K=20 experts resident, loading others from SSD on demand
- Float16 compute path on Apple Silicon via MLX, faster than ternary MPS
- Engine config: K_override=20, cache_gb=8.0, OUTLIER_MMAP_EXPERTS=0, lazy_load=True
- Performance: 1.59 tok/s, 7.91 GB cache RSS peak, 14.04 GB total RSS peak, 5/5 coherent
- MMLU per GB of RAM is the key metric; 14 GB peak is the upper bound
- Smaller tiers: 4B Nano 71.7 tok/s, 9B Lite 53.4, 26B‑A4B Quick 14.6, 27B Core 40.7, 35B‑A3B Vision 64.1, 397B Plus 1.59
- Runtime built with Tauri + Rust + MLX, available at outlier.host