OpenAI Audit Reveals 30% of SWE‑Bench Pro Tasks Are Broken
Review the SWE‑Bench Pro audit findings and adjust evaluation pipelines accordingly.
Review the SWE‑Bench Pro audit findings and adjust evaluation pipelines accordingly.
Summary
OpenAI’s audit of the SWE‑Bench Pro benchmark reveals widespread task issues, estimating that roughly 30 % of the 731‑task public split are broken. The audit used a datapoint analysis pipeline that flagged 200 tasks (27.4 %) and a human annotation campaign that identified 249 tasks (34.1 %). The broken tasks fall into four categories: overly strict tests, underspecified prompts, low‑coverage tests, and misleading prompts.
Frontier models improved from a 23.3 % to 80.3 % pass rate on the 731‑task public split over eight months, but the presence of broken tasks undermines the benchmark’s validity. The audit combined automated filtering, investigator‑agent reviews, and a five‑engineer human annotation campaign, highlighting the difficulty of curating hard but fair benchmarks and the utility of agents for scalable data quality checks.
OpenAI recommends careful examination of evaluation results and adjustment of pipelines. The audit informs OpenAI’s deployment and safety decisions and underscores the need for trustworthy evaluation data. The study was conducted in June 2026 and published on July 8, 2026.
Key changes
- Audit found ~30 % of SWE‑Bench Pro tasks are broken, with 200 tasks flagged by the pipeline and 249 by human reviewers.
- Broken tasks fall into four categories: overly strict tests, underspecified prompts, low‑coverage tests, and misleading prompts.
- Frontier models improved from 23.3 % to 80.3 % pass rate on the 731‑task public split over eight months.
- The audit used a datapoint analysis pipeline, investigator‑agent reviews, and a five‑engineer human annotation campaign.
- The study highlights the difficulty of curating hard but fair benchmarks and the utility of agents for scalable data quality checks.
- OpenAI recommends careful examination of evaluation results and adjustment of pipelines.
- The audit informs OpenAI’s deployment and safety decisions.