What it is
SWE-Bench Pro is a benchmark for testing whether AI coding agents can handle realistic, large-scale software engineering work — not toy problems, but the kind of multi-file, multi-hour task a professional engineer might spend a day on. Given a real codebase and a description of a bug or feature request, the AI agent must produce a code patch that fixes the problem without breaking anything else, which is then checked automatically against a hidden test suite. It was built by researchers including Xiang Deng, Jeff Da, and Edwin Pan, in partnership with the AI data-and-evaluation company Scale AI, and first published as a paper on arXiv in September 2025 (revised in November 2025).
The benchmark is a response to a specific problem: its predecessor, the original SWE-Bench (and its widely used “Verified” subset), had become too easy. Leading AI models were scoring above 70% on SWE-Bench Verified, and researchers suspected some of that success reflected models having effectively memorized solutions from training data, rather than genuine problem-solving. SWE-Bench Pro tries to close that gap with 1,865 tasks pulled from 41 real repositories, split into a public set (731 tasks, openly available), a held-out set (858 tasks, withheld to prevent gaming), and a commercial set (276 tasks, drawn from proprietary codebases via partnerships with startups) that is never published at all. On the public set, top models initially scored only around 23%, compared with 70%+ on the older, easier benchmark — a sign, the creators argued, that the new tasks were appropriately harder and less contaminated.
Benchmarks like this matter beyond academic interest because they function as scoreboards the AI industry uses to make public claims. A company can point to a benchmark score as evidence its model is “better at coding,” and those claims propagate into product marketing, investment narratives, and policy discussions about AI capability trajectories.
Why it matters for AI governance and narratives
SWE-Bench Pro’s arc — hailed as a rigor upgrade, then partially retracted as unreliable — illustrates a recurring dynamic in the AI narrative environment: capability claims are only as trustworthy as the measurement infrastructure behind them, and that infrastructure is contested, proprietary, and often audited by the same labs whose models it evaluates. When OpenAI, a company with an obvious stake in how coding capability is measured, published a detailed critique concluding that roughly 30% of SWE-Bench Pro’s public tasks were defective — and that the benchmark’s dramatic pass-rate climb (23% to over 80% in about eight months) likely reflected the benchmark being “solved” through flaws and overfitting rather than genuine capability gains — it was simultaneously an act of scientific self-correction and a competitive move that reshapes which numbers get cited in the next round of model launches. The episode is a useful test case for the observatory’s broader thread on benchmark legitimacy: who gets to audit the auditors, and what happens to public narratives about AI progress when the yardstick itself turns out to be unreliable.
Key facts and dates
SWE-Bench Pro was first posted to arXiv on September 21, 2025 (arXiv:2509.16941), with a revised version posted November 14, 2025. It is maintained and hosted on a public leaderboard by Scale AI. OpenAI’s audit, published as a blog post titled “Separating signal from noise in coding evaluations,” used a multi-stage process: an automated screening tool flagged 286 suspicious tasks out of the 731 public tasks; a deeper review using OpenAI’s Codex agent narrowed this to 200 tasks (27.4%) confirmed as flawed; a separate, independent human-annotation campaign by five experienced software engineers flagged an even higher share, 249 tasks (34.1%), with roughly 74% agreement between the automated and human reviews. The reported problems fell into recurring categories: tests that were too strict (rejecting functionally correct solutions over unstated implementation details), prompts that were too vague (omitting requirements that hidden tests nonetheless enforced), tests too shallow to catch incomplete solutions, and outright misleading task descriptions.
On the strength of this audit, OpenAI formally retracted its earlier recommendation that the AI research community adopt SWE-Bench Pro as a successor to SWE-Bench Verified, while stopping short of naming a replacement — instead calling on the field to build benchmarks that are harder to game and more trustworthy, with heavier involvement from experienced developers in task validation.
Where to learn more
- SWE-Bench Pro paper on arXiv — the original research paper describing the benchmark’s design, task composition, and initial model results
- OpenAI: “Separating signal from noise in coding evaluations” — OpenAI’s primary audit report and formal retraction of its SWE-Bench Pro recommendation
- SWE-Bench Pro public leaderboard (Scale AI) — the benchmark’s official leaderboard and methodology page
- scaleapi/SWE-bench_Pro-os on GitHub — the open-source repository for the public task set