What it is
“Independent evaluators” refers to outside organizations — separate from the company that built an AI model — that test the model for safety risks before or shortly after it is released. The practice emerged because AI developers evaluating their own systems face an obvious conflict of interest: the company that stands to profit from a fast launch is also the one deciding whether the launch is safe.
In practice, the field is dominated by a handful of nonprofits — METR, Apollo Research, and Redwood Research chief among them — alongside two government bodies, the UK’s AI Security Institute (AISI) and the US Center for AI Standards and Innovation (CAISI, housed at NIST). These groups get pre-release or near-release access to models from OpenAI, Anthropic, Google DeepMind, and others, and run tests for things like autonomous replication, cyberattack capability, deception, and the ability to accelerate AI research itself. METR, for instance, evaluated OpenAI’s o1 models and Claude 3.5 Sonnet for autonomous capabilities; the UK and US AISIs have run joint evaluations of Anthropic and OpenAI models, and more recently of China’s Kimi K3, on cyber and software-engineering benchmarks.
The arrangement has always been uneasy. Evaluators depend on the labs for access, and labs decide how much access to give and for how long. In 2026, Apollo Research reported it was given roughly three days to test OpenAI’s a major release before launch — far short of what researchers say rigorous evaluation requires — and similar complaints have surfaced about compressed timelines and restrictive non-disclosure agreements limiting what evaluators can publish.
Why it matters for AI governance and narratives
… [full content as written]