NVIDIA NeMo AutoModel Accelerates Transformers v5 MoE Fine‑Tuning
Switch to NeMo AutoModel for MoE fine‑tuning to get 3.5× speedup and 30% less memory while keeping the same `from_pretrained()` API.
Import `NeMoAutoModelForCausalLM` and load your MoE model with `from_pretrained()`; enable Expert Parallelism and DeepEP for multi‑GPU training.
Summary
On June 24 2026, NVIDIA announced that the NeMo AutoModel library now builds on HuggingFace Transformers v5, adding Expert Parallelism, DeepEP fused all‑to‑all dispatch, and TransformerEngine kernels to accelerate MoE fine‑tuning. The integration yields 3.4–3.7× higher training throughput and 29–32% less GPU memory compared to native Transformers v5, while keeping the same `from_pretrained()` API. Benchmarks on a 16‑node H100 cluster fine‑tuning Nemotron 3 Ultra 550B achieved 815 TFLOP/s per GPU and 58.2 GiB peak memory, a feat impossible with vanilla Transformers v5. On a single node with 8 H100 GPUs, NeMo AutoModel achieved 3.69× speedup over v5 for Qwen3‑30B‑A3B and 3.36× for Nemotron 3 Nano 30B A3B, with memory reductions of 29% and 32% respectively. The library subclasses `AutoModelForCausalLM`, supports expert backends, dynamic weight loading, and distributed execution via PyTorch DeviceMesh. It also provides custom kernels for fused attention, linear layers, and RMSNorm, and can export standard HF checkpoints for use with vLLM or SGLang. The release positions NeMo AutoModel as a turnkey solution for scaling MoE models across multi‑GPU setups without code changes. Developers can now train frontier‑scale MoE models more efficiently by simply switching to `NeMoAutoModelForCausalLM.from_pretrained()`.
Key changes
- NeMo AutoModel now supports Transformers v5 MoE features: expert backends, dynamic weight loading, distributed execution.
- Adds Expert Parallelism, DeepEP fused all‑to‑all dispatch, and TransformerEngine kernels.
- Achieves 3.4–3.7× throughput increase and 29–32% GPU memory savings over native Transformers v5.
- 16‑node H100 fine‑tune of Nemotron 3 Ultra 550B reaches 815 TFLOP/s per GPU, 58.2 GiB peak memory.
- Single‑node 8 H100 benchmark: 3.69× speedup for Qwen3‑30B‑A3B, 3.36× for Nemotron 3 Nano 30B A3B.
- Supports `from_pretrained()` API with no code changes; exports HF checkpoints.
- Subclasses `AutoModelForCausalLM`, integrates with PyTorch DeviceMesh for distributed training.
- Provides fused attention, linear, and RMSNorm kernels via TransformerEngine.