What it is
ExploitBench is a technical benchmark for measuring how far an AI system can progress through the stages of building a working software exploit — not just finding a bug, but turning it into a functioning attack. It was built by Seunghyun Lee and David Brumley of Carnegie Mellon University, developed with Bugcrowd, and published as a paper on arXiv on May 13, 2026 (“ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents”). It is also released as open infrastructure on GitHub and via a public leaderboard at exploitbench.ai.
Rather than scoring exploitation as a binary pass or fail, ExploitBench treats it as a ladder of 16 discrete, machine-verifiable capabilities grouped into five tiers — from reaching the vulnerable line of code and triggering a crash, through building exploit “primitives” such as arbitrary memory read/write, up to hijacking a program’s control flow and finally achieving arbitrary code execution (ACE), meaning full control of the target. The benchmark tests AI coding agents against 41 real vulnerabilities in Chromium’s V8 JavaScript/WebAssembly engine, all patched but reported from 2024 onward, run against hardened release builds with the same defenses — heap sandboxing, ASLR, stack canaries — present in production browsers. Agents work inside containerized exploitation environments through a standard tool-use interface (MCP), and outcomes are validated deterministically rather than graded by a human or another model.
Why it matters for AI governance and narratives
ExploitBench matters to the observatory because it has become a concrete data point that AI labs and their critics both reach for when arguing about frontier-capability risk — and different actors cite different numbers from the same results table to make different points. Anthropic’s own research posts use the benchmark to argue that its frontier model, Claude Mythos Preview, reaches arbitrary code execution on a majority of the 41 CVEs tested (21 of 41), a result the company frames as evidence that “near-deterministic” exploit generation has arrived as a frontier capability requiring safeguards. Anthropic has also used ExploitBench to warn that Zhipu AI’s openly downloadable GLM-5.3 builds working exploits with comparatively weak safeguards, and that those safeguards erode further under adversarial fine-tuning (“abliteration”) or simple prompting tricks — a framing that casts open-weight releases as a proliferation risk, a recurring motif in how Western labs discuss Chinese model releases.
Coverage of the same underlying results diverges by ecosystem. The editorial passage that prompted this explainer noted that Chinese aggregators reproduced a benchmark line — GLM-5.3 completing 50 of 410 attempts against Claude Mythos Preview’s 56 — that English-language coverage of the story left out. That figure uses a different denominator (total attempts across repeated trials and random seeds) than the “21 of 41 CVEs” figure featured on Anthropic’s own site (CVEs on which a model achieved ACE at least once), and it narrows the apparent gap between GLM-5.3 and Anthropic’s model considerably. Neither figure is wrong; they measure different things. But which metric a given outlet foregrounds shapes whether the story reads as “China closes the capability gap” or “Western frontier models pull decisively ahead” — a clear case of a single technical benchmark supplying raw material for competing capability narratives.
Key facts and dates
The ExploitBench paper was submitted to arXiv on May 13, 2026, by Seunghyun Lee and David Brumley of Carnegie Mellon University, developed with Bugcrowd. The initial release (v0.1) is built on 41 V8 vulnerabilities reported in 2024 or later, decomposed into 16 flags across five capability tiers. The project is open source (MIT license) at github.com/exploitbench/exploitbench, with results published on a public leaderboard at exploitbench.ai; Epoch AI, an independent research organization, also publishes commentary and hosts leaderboard data for it.
Anthropic has used the benchmark in at least two published research pieces: an evaluation of its own Mythos Preview model, which reported ACE on 21 of 41 CVEs versus a next-best competitor reaching ACE on only 2 of 41 (and requiring proprietary scaffolding to do so); and a September 29, 2026 assessment of Zhipu AI’s GLM-5.3, which reported the model completing 50 of 410 individual exploitation attempts, achieving full control-flow hijacks in 4% of trials, and — in a separate demonstration — independently discovering and chaining previously unknown browser vulnerabilities into a working exploit that could read arbitrary files from a visitor’s computer.
Where to learn more
- ExploitBench paper (arXiv): https://arxiv.org/abs/2605.14153
- ExploitBench code and public leaderboard: https://github.com/exploitbench/exploitbench
- Anthropic, “Measuring LLMs’ ability to develop exploits”: https://www.anthropic.com/research/exploit-evals
- Anthropic, “GLM-5.3 and the spread of advanced cyber capabilities”: https://www.anthropic.com/research/glm-5-3-and-the-spread-of-advanced-cyber-capabilities