The agent was measured in somebody else's company

The demo always works. An agent reads the email, finds the right customer, updates the record, drafts the reply, and everyone in the room nods. Then the same agent goes to work on your actual system, and it starts writing delivery dates into the purchase order field, because your team has been abbreviating that field header for years and nobody ever wrote the convention down.

The instinct is to blame the model, or to conclude that agents do not work yet. Neither is quite right. Almost every number you have seen about how well agents perform was produced inside a simulated company: invented tools, an invented policy document, and a scoring rule that decides what counts as done. Your business was not in that simulation. The distance between where the measuring stopped and where your software starts is where production failures live, and it is measurable in an afternoon. This guide is for operators about to wire an agent into a system they would rather not break. If you are building an agent product with a continuous evaluation harness, you want the vendor documentation directly; this is the small-team version.

Every agent score you have seen came from somebody else’s company

It helps to see one of these simulations concretely. τ-bench, published in 2024, emulates dynamic conversations between a user simulated by a language model and an agent given domain-specific API tools and policy guidelines, then compares the database state at the end of the conversation against an annotated goal state [6]. That is a simulated retail company: a written policy, a set of API calls, and a database whose final state defines a correct outcome. The authors built it because, in their words, existing benchmarks “do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications” [6].

None of this is dishonest. It is the only practical way to compare agents at all. But it is a specific company, and it is not yours. The vendors say as much, quite plainly, in the places nobody reads. Anthropic’s computer use documentation lists known limits of the current toolset: Claude “might make mistakes or hallucinate when outputting specific coordinates while generating actions”, “might make mistakes or hallucinate when selecting tools while generating actions”, and reliability “might be lower when interacting with niche applications or multiple applications at once”. On spreadsheets, the same page says “complex spreadsheet operations might still require multiple attempts” [3].

Read that last one again if your business runs on spreadsheets. It is not a warning about a broken product. It is a description of the residue left over when the general case ends and your files begin. Your field names, your three customers who are billed differently, the project code that means one thing in the CRM and another in the invoice template — none of that was in any benchmark, and no amount of general capability infers it.

Running a task once tells you almost nothing

The single most useful idea to steal from the benchmark people is that success is a distribution, not an event. The τ-bench authors proposed a metric called pass^k specifically “to evaluate the reliability of agent behavior over multiple trials”, and their experiments found that even state-of-the-art function calling agents, gpt-4o among them, “succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)” [6]. The headline number aged, as headline numbers do. The finding underneath it did not, and the authors stated it plainly: the results point to a need for “methods that can improve the ability of agents to act consistently and follow rules reliably” [6]. An agent that passes a task can fail the same task on the next attempt, with the same inputs.

This matters more to you than to a lab. A lab can average over a thousand runs. You have one accounts process, and it assumes that whatever happened on Tuesday will happen again on Wednesday. An agent that gets it right four times out of five is not an 80% employee. It is a process that quietly generates one bad record a week, in a system where bad records are discovered by customers.

So run every test task at least 5 times, and write down how many of the runs you would actually have shipped. The arithmetic is in your favour. Claude Sonnet 5 is priced at $2 per million input tokens and $10 per million output tokens [8], so 100 runs that each read 20,000 tokens and write 2,000 come to roughly $6. Budget above that figure rather than at it: the same pricing page notes that Claude 4.7 and later models use a newer tokenizer producing approximately 30% more tokens for the same text [8]. It is still cheaper than one afternoon spent unpicking a corrupted export.

Build the test set out of last month’s actual work

You do not need a simulator. You need about 20 real jobs the agent will be asked to do, taken from work you have already done, where you know what the right answer was.

Anthropic’s guidance on building evaluations is blunt about where the value is: “Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases” [1]. The edge cases it names are the ones that will actually break you: irrelevant or nonexistent input data, overly long input data, poor or irrelevant user input, and ambiguous test cases where even humans would find it hard to reach an assessment consensus [1]. In your business those have names. The invoice with no PO number. The email thread where the client changed the brief in the middle. The 40-page attachment.

The same page pushes against the instinct to build something small and beautiful: “Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals” [1]. For a solo operator that translates to a rule of thumb. Take last month’s folder, pull 20 items, and make sure at least 3 of them are ones that went wrong the first time a human did them.

On grading, the documentation ranks the options. Code-based grading is “fastest and most reliable, extremely scalable, but also lacks nuance for more complex judgments”. Human grading is “most flexible and high quality, but slow and expensive. Avoid if possible.” LLM-based grading is “fast and flexible, scalable and suitable for complex judgment”, with the instruction to “test to ensure reliability first then scale” [1]. Start with whatever you can check with a formula. Whether the right record changed, and no other record changed, is a question a spreadsheet can answer.

Write the pass condition before you run anything

The test set is worthless if you decide what counts as success after seeing the output, because you will grade generously. Anthropic’s guidance on success criteria asks for four properties: specific, measurable, achievable, and relevant [2]. Its own illustration of the difference is worth copying. Bad: “The model should classify sentiments well.” Good: an F1 score of at least 0.85 on a held-out test set of 10,000 diverse Twitter posts, described as a 5% improvement over the current baseline [2].

Your version does not need a statistics vocabulary. It needs to be checkable by someone who was not in the room. “Handles our invoices well” is not checkable. “The supplier name, invoice number, net total and due date match the PDF, no other field on the record is modified, and anything missing from the PDF is left blank rather than guessed” is checkable, and a stranger can grade 20 runs against it without phoning you.

Include more than one dimension. The documentation notes that most use cases need multidimensional evaluation along several success criteria, and lists task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency and price among the common ones [2]. An agent that is accurate but takes 4 minutes per record has failed a different test than one that is fast and wrong, and you want to know which one you bought.

Give it a copy of the system, not the system

The first run of your test set should not touch anything real. This is the one point where vendor documentation, security guidance and hard experience all say the same thing.

Anthropic’s computer use page recommends using “a dedicated virtual machine or container with minimal privileges to prevent direct system attacks or accidents”, avoiding giving the model access to sensitive data such as account login information, limiting internet access to an allowlist of domains, and asking a human to confirm decisions that might result in meaningful real-world consequences. It also runs classifiers that flag possible prompt injections in screenshots, and states that “these precautions remain important even with the classifier defense layer in place” [3]. OpenAI’s computer use guide gives the same instruction in fewer words: “Use an isolated browser or VM and an allow list of sites and actions. Keep access limited to what the task needs” [5]. Anthropic’s engineering write-up on agents is equally direct: “We recommend extensive testing in sandboxed environments, along with the appropriate guardrails” [4].

For a two-person business this is less exotic than it sounds. A duplicate of the spreadsheet. A free test workspace in the same tool. A sandbox account at the vendor. If the agent reaches your systems through Zapier, n8n or Make, point the final step at a test table for the first 20 runs and read what lands in it. The important property is not sophistication. It is that a wrong answer costs nothing to discard.

Set limits while you are there. OpenAI’s guidance is to “Set step, time, or cost limits, support cancellation, and check the actual outcome instead of relying only on the model’s final answer” [5]. An agent with no step cap does not fail, it loops, and you find out from the bill.

Grade the record, not the transcript

When a run goes wrong, the transcript will usually read well. That is the part models are best at. The instruction to check the actual outcome rather than the model’s final answer [5] is the whole discipline in one clause. Open the record. Compare it to the source document. The narration is not evidence.

Log the shape of each failure, not just the count. Refusing to complete a task and completing it incorrectly are the same number in a pass rate and completely different problems in a business. The first costs you a few minutes. The second puts a wrong number into a system that other things read from. Anthropic’s agent guidance names the reason this compounds: agent autonomy brings “higher costs, and the potential for compounding errors” [4]. A wrong lookup at step 3 makes steps 4 through 9 confidently wrong, all of them well-formatted.

After 20 tasks times 5 runs you will have a short list of failure categories rather than a score, and the list is the actionable output. The categories that repeat are the ones worth fixing, and most of them are fixed by giving the agent something it never had: the field conventions, the exceptions list, the one rule everybody in your business knows and nobody has written down.

Re-run the set when the model underneath changes

Your test set is not a one-off exercise, because the thing you tested is replaced on a schedule. Anthropic publishes a four-state lifecycle — active, legacy, deprecated and retired — provides “at least 60 days’ notice before model retirement for publicly released models”, names a recommended replacement when a model is deprecated, and recommends that “to help measure the performance of replacement models on your tasks, consider thorough testing of your applications with the new models well before the retirement date” [7].

The schedule is not theoretical. Claude Opus 4 and Claude Sonnet 4 were retired on 15 June 2026 and Claude Opus 4.1 on 5 August 2026. Of the models still listed as active, Claude Sonnet 4.5 carries the nearest tentative retirement date, given as not sooner than 29 September 2026, followed by Claude Haiku 4.5 at not sooner than 15 October 2026 [7]. If you built a process on one of those, the clock has been running while you were not looking.

The tools move too. Anthropic’s current computer use toolset, computer_toolset_20260801, needs no beta header and supports Claude Fable 5 and 5.1, Mythos 5 and 5.1, Opus 4.8 and 5, and Sonnet 5. Claude Opus 4.7, Opus 4.6, Opus 4.5 and Sonnet 4.6 support computer use only through the earlier computer_20251124 version, which does require a beta header [3]. If your agent is embedded in somebody else’s product, you may not even be told which of these you are on this month.

This is why the 20 tasks are an asset rather than a chore. Building them takes an afternoon, once. Re-running them is the cheap part, and it converts every vendor announcement from a thing you worry about into a thing you check.

What still goes wrong

A copy of your system is not your system. Staging data is cleaner, smaller and better behaved than production data, and the record that breaks the agent is usually the one created in 2018 by somebody who has left. Twenty tasks run 5 times is 100 samples, which is enough to find a failure category that shows up 1 time in 10 and nowhere near enough to find the one that shows up 1 time in 500 and costs you a client. Treat the test set as a filter that catches the obvious, not as a certificate.

Some failures are not in the task distribution at all. Anthropic’s computer use page warns that “jailbreaks and prompt injection can affect computer use as they can any frontier AI system, including through instructions embedded in webpages or images” [3], and OpenAI’s guidance is to treat screen content as untrusted, since text on a page cannot be allowed to override the user’s instructions, and to require user confirmation for consequential actions such as purchases, data transmission and destructive changes [5]. Your 20 invoices will never contain an instruction hidden in a PDF telling the agent to email the file elsewhere. That risk is handled by scoping permissions and by keeping a human on irreversible actions, not by testing harder.

And you are evaluating a moving object with an unmarked schedule. You will not always be told when a product’s underlying model changed, when a system prompt was tuned, or when a tool version rolled forward. The honest position is that a passing test set tells you the agent worked on your tasks on the day you ran it. That is a much weaker claim than “the agent works”, and it is still the strongest claim anyone in this market can make about your business specifically.

Prompts from this guide

agent-test-set-builder

I am about to test an AI agent that will {task_description} inside
{system_name}.

Below are {n} real examples of this task that were completed by a human,
each with the input material and the final correct result.

For each example, write a test case with:
1. The exact input the agent will receive
2. A pass condition a stranger could grade without asking me anything,
   naming the fields or records that must change and the ones that must
   not change
3. The single most likely way an agent could produce a plausible but
   wrong result on this specific example

Then list any edge case you think is missing from this set, based on
what you can see in the examples. Do not invent examples; only tell me
what category is absent.

---
{examples}

agent-run-grader

Grade this agent run against the pass condition. Do not read the agent's
own summary of what it did as evidence of what it did.

Pass condition:
{pass_condition}

Source material the agent was given:
{source_material}

Final state of the record after the run:
{final_state}

Answer in this order:
1. PASS or FAIL
2. If FAIL, the category: wrong value, missing value, invented value,
   touched something it should not have, refused, or incomplete
3. The specific field and the specific difference, quoted from the two
   inputs above
4. Nothing else. No suggestions.