Strix Halo Clustering Discussion for Larger LLMs
Explore clustering Strix Halo nodes with 50G/100G Ethernet and test tensor parallelism via vLLM to evaluate performance before scaling.
Set up a test cluster of two Strix Halo nodes, connect them with 50G Ethernet, install vLLM, and benchmark tensor parallelism on Minimax 2.7 high‑q3 to gauge feasibility.
Summary
The user purchased a Strix Halo (bosgame m5) and observed a 10–20% price hike within two weeks, prompting a plan to cluster two nodes to achieve 256 GB of RAM. With 128 GB, they can run Minimax 2.7 high‑q3, GLM 4.7 q1/q2, and Qwen 3.5 ≈ 400 B; a 256 GB cluster would support Minimax q4 2.7, GLM 4.7 q4, GLM 5.1 q1/2, and Qwen 3.5 q4.
They are evaluating clustering methods such as thunderbolt networking (high latency, no RDMA), M2 NVMe adapters to Oculink, 50 G/100 G Ethernet, and Nvidia DGX spark connectors, and are curious about achievable speeds and latency. Parallelism techniques under consideration include tensor parallelism for prefill acceleration, pipeline parallelism for token‑per‑second gains, Exo compatibility, and the vLLM Strix Halo toolbox.
As an embedded engineer comfortable with hacky solutions, they have already trained models on Strix Halo using PyTorch, albeit with a messy setup, and are seeking documented experiences and recommendations from the community.
Key changes
- Strix Halo (bosgame m5) price rose 10–20% in two weeks
- 128 GB node can run Minimax 2.7 high‑q3, GLM 4.7 q1/q2, Qwen 3.5 ≈ 400 B
- 256 GB cluster can run Minimax q4 2.7, GLM 4.7 q4, GLM 5.1 q1/2, Qwen 3.5 q4
- Clustering options: thunderbolt networking (high latency, no RDMA), M2 NVMe adapters to Oculink, 50 G/100 G Ethernet, Nvidia DGX spark connectors
- Parallelism: tensor parallelism for prefill, pipeline parallelism for token/sec, Exo compatibility uncertain, vLLM Strix Halo toolbox potential
- User trained models on Strix Halo with PyTorch but setup was messy