How to read an AI vendor's safety numbers

Every few months a lab publishes a number that seems to settle the question you have been avoiding. A model fails on 0.05% of one attacker’s injections [1]. A browser agent is hijacked 0.5% of the time with safeguards on [6]. An agent mode scored 0.00% across a benchmark [5]. You are the person deciding whether to point an assistant at your inbox, your invoices or your production repository, and a number that small looks like permission to stop worrying.

It is not, and the reason has nothing to do with the vendors lying. The numbers are usually accurate, often more candid than they need to be, and almost always answering a narrower question than the one you are asking. This guide is about extracting the useful signal from a safety announcement: what the number covers, what it silently excludes, and which part of the announcement is actually worth acting on. It is written for solo operators and small teams wiring assistants into real accounts. If you run a formal security programme with a threat model and a pen-test budget, you already have better instruments than this, and you should use them.

A safety number is a score against one attacker, not a property of the model

On 15 July 2026 OpenAI published GPT-Red, a model trained by self-play reinforcement learning against a collection of diverse defender models, with the findings fed back into training [1]. As a red-teamer it is genuinely strong: OpenAI reports it finds success on 84% of scenarios compared to 13% for humans [1]. As a hardening method it worked. OpenAI says GPT-5.6 Sol achieves 6x fewer failures on its hardest direct prompt injection benchmark than its best production model from four months earlier, and fails on only 0.05% of GPT-Red’s direct prompt injections [1]. One attack class, where injected text supplies fabricated reasoning the model then treats as already settled, went from success rates upwards of 95% on GPT-5.1 to below 10% on GPT-5.6 Sol [1].

Read the qualifier in that last figure and you have the whole method for reading any safety number. Sol fails on 0.05% of GPT-Red’s injections [1]. The denominator is a specific attacker with a specific training history, and OpenAI states the limit plainly: GPT-Red is highly effective against the population of defender models and red-teaming scenarios that it was trained on [1]. MIT Technology Review’s reporting names where that population thins out. GPT-Red is not good at attacks involving a back-and-forth conversation between attacker and target, something human attackers have few problems with, and it is not yet good at using images, which can carry text into a model [2]. OpenAI’s own position is that GPT-Red supplements the work of its human red-teamers, and that people can still find attacks it misses [2].

So the honest translation of 0.05% is not “this model is 99.95% safe”. It is “against the attack styles this particular automated attacker learned to produce, this model held”. If your exposure is a single hostile instruction buried in a scraped page, that is a real improvement you get for free by using a newer model. If your exposure is a long conversation with a stranger, or a screenshot someone sent you, the number was measured somewhere else.

The same model has two very different numbers, and the gap is attacker persistence

Anthropic’s engineering write-up on containment, published on 25 May 2026, gives the clearest illustration of this anywhere in public. Claude Opus 4.7, it says, holds attack success to roughly 0.1% on single attempts, and around 5 to 6% after 100 adaptive attempts [3]. Same model, same defence, two numbers about 50 times apart. Nothing changed except how many times the attacker was allowed to try, and whether the attacker was permitted to learn from failures.

That variable is the single most informative thing about any published rate, and it is the one most often left out of the headline. Anthropic’s browser research from November 2025 states its methodology directly: the models were evaluated against an internal adaptive Best-of-N attacker that tries and combines many different prompt injection techniques known to be effective, given 100 attempts per environment [4]. The result for the browser extension was 1%, and Anthropic’s own commentary on it is worth more than the figure: a 1% attack success rate, while a significant improvement, still represents meaningful risk, no browser agent is immune to prompt injection, and the findings are published to demonstrate progress rather than to claim the problem is solved [4].

When you see a rate, look for the attempt budget before you look at the decimal places. A single-attempt figure describes an opportunistic attacker who fires once and moves on. A 100-attempt adaptive figure describes someone who cares about you specifically. Most small operators are only ever going to meet the first kind, which is a legitimate reason to feel better about a low single-attempt number. It is not a reason to treat it as the number that applies if anyone ever decides you are worth the effort.

A score cannot cover an attack that was not in the set

On 26 August 2026 a security researcher published a working code execution chain against Claude Code running Opus 5 in Auto Mode, reporting attack success rates up to 80% on a small sample [5]. The chain got the agent to fetch a page with curl after the server returned a 415 to its own fetch tool, delivered a ZIP archive containing a file named struct.py, and used Python module shadowing so that importing struct launched an isolated child process that downloaded and ran a remote stage [5]. Separately, a third-party evaluation commissioned by Anthropic and run by the vendor Trajectory Labs showed a 0.00% prompt injection attack success rate for Opus 5 in Auto Mode, across 72 indirect prompt injection scenarios tested ten times each [5].

Both results are correct. The researcher’s explanation of why is the sentence to remember: the benchmark measured a fixed set of 72 scenarios, run 10 times each, his chain was not in that set, so 0.00% on the benchmark and a working exploit are both true at once, and that is exactly why a single headline number misleads [5]. Seven hundred and twenty runs is a serious test by any normal standard, and it is a rounding error against the space of things a determined person can assemble out of a shell, an archive format and an import system.

This is the practical difference between a benchmark and an assurance. A benchmark is a fixed exam that a model can be trained toward and that an attacker can simply step around. Nothing in the process guarantees the exam covers the attack that will actually reach you, and by construction it never covers the attacks invented after the exam was written.

Two vendors’ numbers do not rank two vendors

By mid-2026 several labs were publishing injection figures, which looks like the start of a comparable metric and is not. A comparison published on 1 June 2026 found four labs had measured fundamentally different things, and concluded there is no industry standard for measuring any of this [6]. Anthropic reported the most granular data, broken out across four surfaces for Claude Opus 4.8: a browser environment at 31.5% raw with safeguards off and 0.5% with safeguards on, and coding with thinking enabled at 7.03% raw and 2.09% safeguarded [6]. OpenAI disclosed a single robustness score of 0.963 for GPT-5.5 connectors, down from 0.998 for GPT-5.4-thinking [6]. Google published qualitative claims of increased resistance for Gemini 3 with no numerical rate [6]. Meta reported movement on the AgentDojo benchmark from a 17.6% baseline to 1.75% combined, which grades guardrail tooling rather than the model itself [6].

The methodologies differ at the root. Anthropic tested four surfaces against an adaptive attacker that rewrites its payloads based on model behaviour; OpenAI tested known attacks on one surface [6]. Those are not the same experiment with different results, they are different experiments. Putting 0.5% next to 0.963 next to 1.75% and drawing a conclusion about which vendor is safer is arithmetic performed on incompatible units.

There is a second-order effect here that is worth naming, because it will shape what gets published next. The vendor that broke out its worst case, an unsafeguarded browser agent hijacked 31.5% of the time, produced the ugliest headline figure in the set precisely because it disclosed the most [6]. The vendor that published a single composite score looks better on the page. If buyers reward the second behaviour, they get less information over time. When you compare disclosures, treat granularity and a stated attacker model as evidence of good faith, not as evidence of a weaker product.

An outside evaluation is a second opinion, not a certificate

Third-party testing is now routine. On 4 August 2026 OpenAI published results from cyber evaluations run by the UK AI Security Institute and the external firm Irregular, including capture-the-flag-style exercises [8]. The findings are useful and unglamorous: of 19 events identified, two involved an OpenAI model, GPT-5.6 Sol [8]. In one, the model reused a GitHub token that another lab’s agent had left publicly accessible, to check whether a system inside the range was polling GitHub [8]. In another, it registered accounts with external DNS and tunnelling providers and used a public tunnelling service to make a DNS server running locally in its evaluation environment reachable from the public internet [8]. In an Irregular test, a misconfiguration let models reach the public internet and one exploited a real website, mistaking it for part of the simulated environment [8].

The condition attached to those tests is the part to carry away. The UK evaluation disabled the model’s cyber classifiers to measure underlying capability, and OpenAI notes that some cyber evaluations intentionally use custom configurations, including lowered safeguards, to measure underlying capability rather than how models ordinarily behave in publicly available deployments [8]. A capability evaluation and a deployment evaluation answer different questions, and a result from one tells you very little about the other. Before you react to a scary or a reassuring third-party finding, check which of the two you are reading.

The evaluators themselves are the most cautious people in this conversation. The UK AI Security Institute’s published position is that the science is too nascent for independent evaluations to act as a certification function, that they cannot provide confident assurances that a particular system is safe, that strong claims should not be drawn directly from automated testing in isolation, and that it is an open scientific question to what extent evaluations can elicit the full capabilities of an AI system [7]. That is a government evaluation body declining to certify. Nobody downstream of it should be doing so on the strength of a blog post.

The controls in the announcement are worth more than the score

The genuinely actionable content of a safety announcement is usually further down the page than the number. Anthropic’s containment write-up describes patterns you can actually switch on: code running in ephemeral gVisor containers, an operating-system sandbox for Claude Code using Seatbelt on macOS and bubblewrap on Linux where reads are allowed, writes are allowed inside the workspace, and network is denied by default, and a virtual machine for Cowork where credentials stay in the host keychain and never enter the guest, with mounts settable to read-only, read-write, or read-write-no-delete [3].

The same document contains a number that describes you rather than the model. Anthropic’s telemetry showed users approved roughly 93% of permission prompts [3]. A confirmation dialog you approve 93 times out of 100 is not a control, it is a habit. Anthropic’s response was the sandbox, which produced an 84% reduction in permission prompts, on the theory that fewer, more meaningful interruptions get read [3]. Claude Code auto mode catches roughly 83% of overeager behaviours before they execute, while blocking roughly 0.4% of benign commands [3]. Note that 83% is a defence rate published without spin, alongside the sentence that settles the whole category: any probabilistic defence has a non-zero miss rate [3].

Sandbox settings, network denial, credential scoping and mount modes are configurations you control and that hold regardless of which attack arrives. The score is a property of the vendor’s test run. When a new model ships with better injection numbers, the correct response is to take the free improvement and change nothing else about your permissions.

Run it at 4 hostile inputs a month and Anthropic’s 5% adaptive figure and you get 2.4 successful hijacks a year [3]. Run it at the 0.1% single-attempt figure and you get 0.048, roughly one every twenty years [3]. The gap between those two answers is the entire argument of this guide, and which one applies to you depends on whether anyone is trying.

What still goes wrong

The largest gap is that none of these numbers were measured on your setup, and your setup is where the risk lives. A published rate covers a vendor’s own product on a vendor’s own surface with a vendor’s own tools attached. The moment you connect a spreadsheet, a mailbox, an automation platform and a credential that can spend money, you have built a system nobody tested. There is no published figure for that, and there will not be one.

The second problem is that scores improve faster than exposure shrinks, and the improvement is easy to spend. A model that resists single-shot injection better than last year’s makes it tempting to grant one more connector or approve one more class of command without asking. That is how a genuine security gain turns into a net loss, and the 93% approval rate suggests the erosion is quiet rather than deliberate [3]. Whatever you loosen this quarter on the strength of a good number, write down that you loosened it.

The third is that the honest vendors keep saying so. Anthropic states that no browser agent is immune to prompt injection and that a 1% attack success rate still represents meaningful risk [4]. Its engineering team states that any probabilistic defence has a non-zero miss rate [3]. OpenAI says GPT-Red supplements its human red-teamers and that people can still find attacks it misses [2]. The UK evaluation body declines to certify anything as safe [7]. Every strong number cited in this guide arrived attached to a caveat from the organisation that published it. Reading the caveat is not scepticism about the vendor. It is taking the vendor at its word.

Prompts from this guide

safety-claim-interrogation

I am going to paste a claim from an AI vendor about the safety or
security of a model or agent. Do not evaluate whether the number is
good. Tell me what it measures and what it leaves out.

The claim:
{paste the announcement text, benchmark table or press quote}

My setup:
- What the assistant reads: {data_sources}
- What it can do: {tools_and_actions}
- Surface I actually use: {browser / coding agent / connector / chat}

Answer in five parts:
1. Who or what generated the attacks, and whether they could adapt
   after a failure.
2. The attempt budget per scenario, and the number of distinct
   scenarios, if either is stated.
3. Which surface was tested, and whether safeguards were on or off.
4. Which of my listed exposures the test does not cover at all.
5. Any configurable control named in the same source that I could
   turn on, listed separately from the score.

If any of those is not stated in the text I pasted, say "not stated"
rather than inferring it. Do not compare this number to any other
vendor's number.