What a test of an AI tool actually proves

You tried the new model on a job from your actual work, read the output, and it was good. So the tool went into the week: client emails, invoice summaries, the first draft of everything. A month later it produced something wrong in front of someone who mattered, and you had no way to say whether that was rare or normal, because the only evidence you ever collected was one output you liked on a Thursday afternoon.

That gap between the trial and the work is the subject here. It is the same gap that the largest AI labs got wrong in public in 2026, at a scale that makes the shape easy to see. This guide is for a solo operator, freelancer or small team deciding whether to hand a recurring job to an AI tool, and wanting the decision to be based on something. It is not a model validation programme for a regulated deployment, it does not cover testing models you train yourself, and it will not tell you which tool to buy.

The number describes the setup, not the model

On 21 July 2026 OpenAI disclosed that models running its internal cyber benchmark got out of the environment built to hold them and reached Hugging Face’s production infrastructure [1]. Set the intrusion aside and read the setup. The evaluation ran in “a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries”, and, in OpenAI’s words, “deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities” [1]. The benchmark itself, ExploitGym, “prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities” [1]. One of the two models involved was “an internal-only research prototype and was never intended for public release” [1].

None of that resembles a model answering your email. That is the point, and OpenAI says so directly about a separate pair of incidents in third-party evaluations: “Some cyber evaluations intentionally use custom configurations, including lowered safeguards to measure underlying capability”, and “The incidents did not reflect ordinary deployment” [2]. Anthropic reached the same place from the other direction. Reviewing 141,006 evaluation runs “where Claude could have obtained internet access”, it identified 3 incidents in which a model reached the internet from inside the evaluation, and noted that “a capture-the-flag scenario is by design an instruction to attack an external system” [3].

So a test result is never a fact about a model on its own. It is a fact about a model under conditions, and the conditions do most of the work. The labs made their conditions unlike deployment deliberately, to measure a ceiling. Your own trial is usually unlike your ordinary week by accident, which is worse, because you did not write down the difference and cannot correct for it.

Vendor scores come with a settings line

The published benchmark numbers you use to pick a tool carry the same clause, printed small. Anthropic’s footnote for its Frontier-Bench v0.1 results with Claude Opus 5 reads: “These results are from an internal run of Frontier-Bench v0.1, on the mini-SWE-agent harness and a GKE backend, mean reward over 5 attempts per task. Opus 4.8 served as fallback on safety-classifier refusals for Opus 5 and Fable 5” [4].

Three conditions in one sentence, and you will reproduce none of them. The run used a specific agent harness, which is scaffolding around the model that you do not have. It reports a mean over 5 attempts per task, while you get one attempt and keep whatever arrives. And a different model, Opus 4.8, stood in when a safety classifier refused, so the reported score is not the score of one model working alone. Elsewhere on the same page, the CursorBench 3.2 result is quoted at max effort, where Opus 5 “performs within 0.5% of Fable 5’s peak score, but at half the cost per task” [4]. Effort is a setting with a bill attached. Anthropic describes it as the control customers “use to optimize for intelligence or conserve tokens for faster and cheaper results”, and Opus 5 is priced at “$5 per million input tokens and $25 per million output tokens” [4].

Read the footnote before the headline, then translate it. The vendor measured with a custom harness, several attempts, the highest effort setting, and a fallback. You will run one attempt, at default settings, in a chat window, on your own messy input. Both are real measurements. They are measurements of different things, and only one of them predicts your Tuesday.

Test on the work you already have, including the ugly two

Anthropic’s guidance on building evaluations puts task specificity first: design evals that “mirror your real-world task distribution”, and factor in edge cases [5]. The edge cases it names are the ones you would never choose for a demo: irrelevant or nonexistent input data, overly long input, poor or harmful user input, and ambiguous test cases “where even humans would find it hard to reach an assessment consensus” [5].

In practice that means your test set is not invented. It is last month’s actual jobs, pulled out of the folder they already live in, with the two horrible ones included. The horrible ones are the whole point. A tool that handles your clean cases and fails your messy ones is a tool that works precisely when you did not need help.

The recommended volumes are larger than you will manage, and worth knowing anyway as a sense of scale. Anthropic’s examples run to 1,000 tweets for a sentiment task, 200 articles for summarisation, 100 inquiries for a customer service tone check, and 50 groups of paraphrased questions to test whether an FAQ bot answers consistently [5]. Its stated preference is blunt: “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals” [5]. You are not going to assemble 200 articles this afternoon. You can assemble 12 real jobs, and 12 graded cases is a different kind of evidence from one output you liked.

One clean run is not a result

The same prompt does not reliably produce the same output, and this is a property of how the models are served rather than a setting you forgot. Researchers at Thinking Machines Lab sampled 1,000 completions at temperature 0 from an identical prompt with the open-weight Qwen/Qwen3-235B-A22B-Instruct-2507 model and generated “80 unique completions, with the most common of these occuring 78 times”; the first divergence arrived at the 103rd token [8]. Their explanation is that inference kernels are not batch-invariant, so “the load (and thus batch-size) nondeterministically varies” between requests and the arithmetic comes out slightly differently [8]. Even at temperature 0, they write, LLM APIs “are still not deterministic in practice” [8].

Two consequences for a small trial. First, run each test case more than once, ideally at different times of day, because a busy hour and a quiet hour are not the same server load, and load is the variable that moves the output [8]. Second, be honest about what a clean run of a small set can detect. If a tool fails 1 time in 20 and you run 10 cases once each, the expected number of failures you see is 0.5, so the most likely outcome of your trial is a perfect score on a tool that will embarrass you twice a month.

Decide what passing means before you look at the output

OpenAI’s definition is the useful one to steal: evals “test model outputs to ensure they meet style and content criteria that you specify” [6]. The criteria are yours, and they exist before the run, because criteria written after you have read a good output are a description of that output.

The mechanics scale down cleanly. OpenAI’s evals require test data pairing each input with a human-provided ground truth label to compare against, plus a grader that decides correctness, of which the simplest kind is a string check against a reference value [6]. For your 12 cases that is a spreadsheet with 3 columns: the input, what a correct answer must contain, and pass or fail. Anthropic’s ranking of grading methods explains why to keep it that dumb. Code-based grading is “Fastest and most reliable, extremely scalable, but also lacks nuance”, LLM-based grading is fast and flexible but you “Test to ensure reliability first then scale”, and human grading is “Most flexible and high quality, but slow and expensive. Avoid if possible” [5]. Reading 36 outputs by hand is the trial you will do once and never repeat, which makes it useless for the part that comes next.

Keep the criteria in a file you own rather than inside a vendor’s product. OpenAI is retiring the Evals platform itself: it “will become read-only for existing users on October 31, 2026”, and is “scheduled to shut down on November 30, 2026” [6]. A spreadsheet outlives that.

Whatever you tested is going to be replaced

The tool you approved is a moving object. Anthropic provides “at least 60 days’ notice before model retirement for publicly released models”, and the recent list is not theoretical: claude-opus-4-1-20250805 was retired on 5 August 2026 with claude-opus-4-8 named as its replacement, and claude-sonnet-4-20250514 was retired on 15 June 2026 in favour of claude-sonnet-4-6 [7]. The documentation’s own instruction is to “Test your applications with newer models well before the retirement date of your current model” [7], which only works if the test still exists as a file rather than as a memory of a good afternoon.

This is also the argument for keeping the test set after the decision is made. OpenAI describes writing evals as “an essential component to building reliable applications”, “especially when upgrading or trying new models” [6], and the retirement date is set by the vendor rather than by you [7]. A saved set of 12 real cases and their pass criteria turns a forced migration into 30 minutes of rerunning, and it turns the vaguer worry that the tool got worse last week into a number you can check.

What still goes wrong

A trial of a dozen cases has wide error bars, and no amount of care narrows them. If the true failure rate is 5%, a 12-case, 3-run trial expects fewer than 2 failures, so seeing 0 and seeing 3 are both ordinary outcomes of the same tool. That is a limit of the sample size and not of your method. Treat a small trial as capable of catching gross unsuitability, which it does well, and incapable of certifying reliability, which it never does.

The second limit is that your criteria and your inputs both drift. The pass criteria you wrote in September encode what mattered in September. The inputs get worse when you are busy, which is exactly when you stop checking, and a tool that passed on last month’s tidy work has not been tested on the version you produce at 11pm. Nothing in a trial covers what a model does with untrusted content someone else wrote, either, which is a different problem with different controls.

The last limit is the one the disclosures illustrate. Two labs with dedicated safety and security teams misjudged the conditions of their own evaluations nine days apart. On 21 July 2026 OpenAI reported that its models breached a partner’s production systems from an environment it had described as highly isolated [1]. On 30 July 2026 Anthropic reported 3 incidents, out of 141,006 evaluation runs “where Claude could have obtained internet access”, in which a model reached real systems after an evaluation prompt that “stated explicitly that Claude had no internet access” [3]. In both, the models did what the task pointed at, and the environment turned out not to match the description in the plan. Your test conditions are described in a document too. Assume it is slightly wrong, and prefer a result you can check again next month over a result you only get once.