OpenMythos: Open‑Source Implementation of the Claude Mythos Recurrent‑Depth Transformer
Install open‑mythos via pip, set attn_type to gqa or mla, and run the 3B training script on FineWeb‑Edu while monitoring spectral radius to ensure ρ(A) < 1.
Install open‑mythos, set attn_type, run the 3B training script, and monitor spectral radius to ensure ρ(A) < 1.
Summary
OpenMythos is an open‑source implementation of the Claude Mythos model that uses a Recurrent‑Depth Transformer (RDT) architecture consisting of a Prelude, a looped Recurrent Block, and a Coda.
The model supports two switchable attention types: GQAttention (gqa) with Flash Attention 2 for efficient KV‑head handling, and MLAttention (mla) that caches a compressed KV latent via KV‑lora. Pre‑configured variants ranging from 1 B to 1 T parameters are available through the mythos_1b … mythos_1t helpers, and a training script for the 3B model on FineWeb‑Edu is provided. Training uses AdamW, bfloat16 on H100/A100 GPUs, float16+GradScaler on older GPUs, a linear warmup to cosine decay schedule, and targets 30 B tokens. Stability is guaranteed by the Parcae architecture, which enforces a spectral radius ρ(A) < 1 on the recurrent matrix, preventing residual explosion. Inference depth is controlled by the number of loops, allowing deeper reasoning without increasing parameter count. The package can be installed via pip, with optional flash support using pip install open‑mythos[flash]. Documentation includes a full API reference, dataset guidance, and examples for generating and training the model.
Key changes
- OpenMythos implements a Recurrent‑Depth Transformer with Prelude, Recurrent Block, and Coda stages
- Two switchable attention types: GQAttention (gqa) with Flash Attention 2 and MLAttention (mla) with KV‑lora compression
- Pre‑configured variants from 1 B to 1 T parameters via mythos_1b … mythos_1t
- Training script for 3B model on FineWeb‑Edu uses AdamW, bfloat16 on H100/A100, float16+GradScaler on older GPUs, linear warmup to cosine decay
- Parcae architecture enforces spectral radius ρ(A) < 1 to guarantee stability
- Inference depth is controlled by loop count, enabling deeper reasoning without extra parameters
- Package installation via pip with optional flash support (pip install open‑mythos[flash])
- Documentation provides full API reference, dataset guidance, and usage examples