Parameter Golf: Lessons from 1,000+ participants, 2,000+ submissions, and an open machine learning challenge shaped by coding agents
Use AI coding agents to submit compact models under 16 MB, 10‑minute training on 8×H100s, and evaluate on FineWeb dataset.
Explore: Run a local Parameter Golf style benchmark using Codex to generate and test small models, and analyze agent‑generated code for compliance.
Summary
Parameter Golf challenged participants to build a model under 16 MB, 10‑minute training on 8×H100s, and evaluate on the FineWeb dataset.
Over eight weeks, the competition received more than 2,000 submissions from 1,000+ participants, with agents lowering experimentation cost and increasing participation.
Notable submissions included GPT‑Q‑lite quantization, a new CaseOps tokenizer, XSA attention variant, SmearGate, and mini‑depth recurrence, all pushing the limits of the rules.
Codex‑based triage bots flagged submissions for human review, handling the volume of entries.
RunPod’s $1 M compute sponsorship made the challenge accessible to many.
Agents introduced new challenges for attribution and scoring, as many submissions were incremental improvements on top‑scoring models.
The competition demonstrated that AI agents can accelerate research and highlight creative approaches that might otherwise be overlooked.
Key changes
- Challenge required <16 MB artifact (weights + code) and 10‑minute training on 8×H100 GPUs.
- Over 2,000 submissions from 1,000+ participants, with agents lowering experimentation cost.
- Notable submissions: GPT‑Q‑lite quantization, CaseOps tokenizer, XSA attention variant, SmearGate, mini‑depth recurrence.
- Codex‑based triage bot flagged submissions for human review to manage volume.
- RunPod sponsorship of $1 M compute enabled broad participation.
- Agents introduced new challenges for attribution and scoring.
- Competition highlighted creative approaches beyond incremental improvements.
- Results showed AI agents can accelerate research and surface novel ideas.