Zyphra Unveils ZAYA1‑8B: 1‑Billion‑Parameter MoE Model Outperforms Claude on Math Benchmarks
Integrate ZAYA1‑8B into your MoE inference pipeline to gain high‑reasoning performance with under one billion active parameters, and benchmark it against DeepSeek‑V3.2 or GPT‑5‑High to confirm the trade‑off.
Integrate ZAYA1‑8B into your MoE inference pipeline and benchmark it against your current models to evaluate the performance‑to‑parameter trade‑off.
Summary
Zyphra has announced the release of ZAYA1‑8B, a mixture‑of‑experts (MoE) language model that, despite having only about 1 billion active parameters, matches DeepSeek‑R1 on mathematical reasoning tests while outperforming Claude Sonnet 4.5 on a range of benchmarks. The 8.4‑billion‑parameter model was trained end‑to‑end on a 1,024‑node AMD Instinct MI300X cluster, demonstrating that AMD hardware can rival the NVIDIA‑centric AI ecosystem. ZAYA1‑8B incorporates a custom attention mechanism and a Markovian RSA inference method that generates parallel reasoning traces, boosting scores on AIME, HMMT, and LiveCodeBench.
The model is available through Zyphra Cloud as a serverless endpoint and on Hugging Face under an Apache 2.0 license. Local deployment requires Zyphra’s forked version of vLLM, as the standard vLLM does not support the custom attention architecture. While ZAYA1‑8B excels at math, coding, and reasoning tasks, it falls short on agentic tasks such as reliable function calling and multi‑step instruction following, as evidenced by lower scores on BFCL‑V4 and IFBench. Researchers interested in test‑time compute methods will find the Markovian RSA implementation noteworthy, and the AMD training path offers a compelling alternative to NVIDIA‑only pipelines.
Zyphra’s announcement highlights the growing diversity of hardware platforms in AI development and underscores the importance of efficient model architectures. By achieving state‑of‑the‑art performance on challenging math and coding benchmarks with fewer active parameters, ZAYA1‑8B positions itself as a competitive option for developers seeking high‑quality reasoning capabilities without the cost of massive parameter counts.
Key changes
- Released ZAYA1‑8B, a reasoning‑focused MoE model trained on AMD hardware.
- Active parameter count <1 B, achieving math and reasoning performance comparable to DeepSeek‑V3.2 and GPT‑5‑High under test‑time compute.
- Lightweight expert architecture keeps memory footprint low while maintaining strong reasoning capabilities.
- Suitable for edge or low‑GPU‑memory deployments due to compact size and efficient expert routing.
- Available for download via the author's repository and compatible with MoE inference engines.