Autoresearch Goes Global: Distributed Agents Find 333 Experiments and 17‑Hour Breakthroughs
Patch your training pipeline to evaluate val_bpb after each 5‑minute run and keep only improvements, discarding any experiment that does not lower the metric.
Patch your training pipeline to incorporate val_bpb evaluation and an automated keep/discard loop that runs every 5 minutes on a single GPU.
Summary
Karpathy’s open‑source autoresearch script, released on March 8, was run by 35 autonomous agents across a peer‑to‑peer network of H100s, consumer GPUs, and CPU‑only laptops, completing 333 experiments without human supervision in 17 hours. The distributed agents discovered known‑good techniques such as RMSNorm, tied embeddings, and specific initialization patterns, proving that constrained hardware drives creative search. Each experiment ran for exactly five minutes, and the loop evaluated results with validation bits per byte (val_bpb), keeping only improvements and discarding failures automatically. The pattern demonstrated that training from scratch on a single GPU can outperform larger, manually tuned models when guided by a precise evaluation metric. The distributed approach also revealed that different hardware constraints lead to diverse search strategies, converging on the same high‑quality solutions. The community has already merged many of the discovered optimizations back into production nanochat codebases, showing the practical impact of automated experimentation. Autoresearch’s success underscores that the bottleneck to scaling automated loops is not compute but the quality of the evaluation infrastructure. The pattern can be extended beyond language models to marketing, supply‑chain, and drug‑discovery pipelines where a clear metric exists.
Key changes
- Distributed 35 agents ran 333 experiments in 17 hours across H100s, consumer GPUs, and CPUs
- Discovered RMSNorm, tied embeddings, and specific initialization patterns
- Each experiment lasted 5 minutes and was evaluated with val_bpb
- The loop kept only improvements and discarded failures automatically
- Training from scratch on a single GPU outperformed larger, manually tuned models
- The pattern can be applied to marketing, supply‑chain, and drug‑discovery pipelines
- Many discovered optimizations merged back into production nanochat codebases
- The bottleneck to scaling automated loops is evaluation infrastructure, not compute