Granite 4.1 LLMs: How They’re Built

Granite 4.1 is a family of dense, decoder‑only LLMs (3B, 8B, and 30B) released under an Apache 2.0 license, trained on approximately 15 trillion tokens with a 512 K‑token context window. The models are refined with 4.1 million high‑quality supervised fine‑tuning samples and a multi‑stage reinforcement learning pipeline that uses on‑policy GRPO with DAPO loss. Architectural choices include Grouped Query Attention, Rotary Position Embeddings, SwiGLU activations, RMSNorm, and shared input/output embeddings. Training follows a five‑phase pipeline: foundational pre‑training on 10 T tokens, math/code pre‑training on 2 T tokens, high‑quality data annealing on 2 T tokens, refinement on 0.5 T tokens, and long‑context training that extends the context from 4 K to 512 K tokens. The long‑context phase uses 80 % books and 20 % code data for 8B and 30B models and merges models after each stage to preserve short‑context performance.

Granite 4.1’s RULER benchmark scores show strong performance across 32K, 64K, and 128K contexts, with the 30B base model achieving 85.2, 84.6, and 76.7 respectively. The release includes detailed data‑engineering, supervised fine‑tuning, and reinforcement learning documentation, making it a comprehensive resource for building high‑quality small language models.

The announcement positions Granite 4.1 as a powerful, open‑source alternative for enterprises and developers seeking dense models with long‑context capabilities and advanced instruction following.