ProgramBench: Can Language Models Rebuild Programs From Scratch?
Explore ProgramBench to benchmark your LMs on end‑to‑end program synthesis; evaluate against 200 tasks covering CLI tools, FFmpeg, SQLite, PHP, and note that current models only achieve 95 % pass on 3 % of tasks.
Review ProgramBench and integrate it into your evaluation pipeline.
Summary
ProgramBench is a new benchmark designed to evaluate language models’ ability to create complete software projects from scratch. It contains 200 tasks that span simple CLI utilities to complex systems such as FFmpeg, SQLite, and the PHP interpreter, and it relies on agent‑driven fuzzing to generate end‑to‑end behavioral tests without prescribing implementation structure.
The study evaluated nine different language models and found that none of them could fully solve any of the tasks. The best performing model managed to pass 95 % of the tests on only 3 % of the tasks, and all models tended to produce monolithic, single‑file implementations that diverged sharply from human‑written code.
These results highlight the current limitations of LLMs in holistic software engineering and suggest that significant research is still needed before models can reliably build production‑ready codebases from documentation alone.
Key changes
- Introduces ProgramBench benchmark for end‑to‑end program synthesis from scratch
- Consists of 200 tasks ranging from CLI tools to FFmpeg, SQLite, PHP interpreter
- Uses agent‑driven fuzzing to generate behavioral tests without prescribing implementation structure
- Evaluated 9 language models; none fully solved any task
- Best model achieved 95 % test pass on only 3 % of tasks
- Models tend to produce monolithic single‑file implementations diverging from human code