Only hand an agent work you can check

You give an agent a job that takes it half an hour. It comes back with something long, tidy and confident. You read it, and it reads correct. That is the only signal you have, and it is the one signal a system built to produce plausible text will give you whether the work is right or not. Reading it a second time does not help. A better model does not help either, because the thing your eyes are testing is fluency, and fluency was never the part in doubt.

The labs that publish results they expect people to attack do not rely on reading either. They attach something that can say no. This guide is about copying that move at your scale: deciding which of your work has a check that can fail, building it before the run instead of after, and knowing what to do with the work that has no check at all. It is not for someone typing a question into a chat window and reading the answer. You are the check there, you are looking at the whole output, and that is fine. It is for work you hand off and walk away from.

The machine check is what made the result reportable

On 10 August 2026 Anthropic published what an unreleased research version of Claude did with the Riemann hypothesis, and the shape of the run is worth having in your head. It worked across two sessions in Claude Code, coordinated about 60 subagents, ran 2,400 shell commands and used 31 million output tokens [1]. Its first 650 ideas produced nothing, and a staff member prompted it to try again [1]. The second attempt took about a day and a half, and the result was not a proof of the hypothesis: it raised a lower bound on the fraction of zeros of the Riemann zeta function that satisfy the hypothesis, from 41.6% to 67.2% [1].

Then comes the part that matters more than the number. Claude also produced a formally verifiable proof of its result, and worked with a member of staff to produce a Lean formalization of it [1]. Lean is an open-source programming language and proof assistant that enables correct, maintainable, and formally verified code [3]. Its relevant property here is blunt: it accepts a chain of reasoning or it rejects it, and it has no opinion about how the argument reads. Alongside that, subagents ran thousands of numerical checks against known zeta zeros and refereed one another’s work, searched for counterexamples, downloaded 54 papers from arXiv to confirm the finding was new, and independently re-proved it from scratch [1]. Two of Anthropic’s own mathematicians examined the work afterwards [1].

OpenAI’s write-up of ten advances in mathematics and theoretical computer science, produced by an internal version of Astra, which it calls its next major model, does the same thing. The model formalized each argument in a Lean certificate, and the certificates were published alongside the results [2]. The compute involved was not the headline: OpenAI puts the tokens needed to find all ten solutions at roughly $2,000 at Sol API rates [2].

Strip the Lean file out of either story and what remains is a model saying something. Neither lab asked anyone to trust the transcript, and neither should you. The verification was not a footnote on the result. It was the reason there was a result.

Sort your work by whether a wrong answer announces itself

Almost everything you might hand to an agent falls into one of three groups, and the groups behave completely differently.

In the first, a wrong answer fails on its own. Code with a test suite that already runs. A script whose output you can regenerate. A set of numbers that has to reconcile to a total you knew before the run started. A data transformation where you can pull ten source rows and compare. You do not have to be smarter than the model to catch it being wrong; something else catches it, and it catches it the same way every time.

In the second, nothing fails automatically, but every claim points at something outside the model. A summary of a document you are holding. A research task where each finding carries a URL. A question about a vendor policy where the answer quotes the policy. There is a check here, it is just made of your attention rather than a command: open the source, compare, cross the line off. This is where most useful agent work actually lives, and it is where people fool themselves most often, because output in this group looks exactly like output in the first group.

In the third, there is no external referent at all. Positioning, tone, which of three plans to pick, whether an email will land. Nothing can fail because nothing has been asserted that reality could contradict. That does not make the work useless. It makes it work you have to own personally, and it means the length and confidence of what came back tell you nothing.

The practical move is to say which group a task is in before you start it, out loud, in one sentence. Most people never do this, which is why they end up applying group-one trust to group-two output.

Write the check before the run, not after

OpenAI’s guidance on evaluating model output is explicit about the order: you begin by specifying how the system should behave before implementing and testing the system, and a proper evaluation compares against a ground truth output that the model should match, provided by a human [5]. Anthropic’s guidance on success criteria is the same instruction in different words. Criteria should be specific and measurable, and the docs contrast a bad criterion, “the model should classify sentiments well”, with a good one that names a metric, a test set and a baseline [4]. The docs also tell you not to forget edge cases, and list the ones people skip: irrelevant or nonexistent input data, input that is far too long, and ambiguous cases where even humans would struggle to agree [4].

The ordering is not bureaucracy. Once you have read an output that sounds right, you will grade it kindly, and you will invent the standard that it happens to meet. Writing the check first removes that option. It also produces a useful failure of its own: if you sit down to define what a wrong answer would look like and cannot, you have just discovered the task is in group three, and you should stop planning to leave it unattended.

Keep the check small. One command, one comparison, one document to hold the output against. A check you will actually run beats a review process you will skip on the third run.

A check that shares the model’s assumptions is not a check

Look again at what Anthropic’s subagents were told to do: re-prove the result from scratch, hunt for counterexamples, and search 54 papers for prior work [1]. Every one of those is independent of the argument being tested. None of them is “read this again and tell me whether it is right”.

That distinction is the whole game. Anthropic’s own list of hallucination techniques includes running the same prompt several times and comparing outputs for inconsistencies [6]. That is worth doing, and it is worth being precise about what it buys you: disagreement between runs is a real warning, while agreement between runs tells you the model is consistent, which is not the same thing as correct. Two runs of the same system, given the same material, share the same blind spots.

The evaluator-optimizer pattern, where one call generates and another evaluates in a loop, is described by Anthropic as particularly effective when there are clear evaluation criteria [7]. The condition carries the weight. With criteria, the evaluator is applying a standard you wrote. Without them, it is a second opinion from the same source, delivered in the same voice.

For group-two work, the strongest cheap check is the one Anthropic spells out. Have the model cite quotes and sources for each claim, then have it verify each claim by finding a supporting quote after the response is generated, retracting anything it cannot support [6]. For documents over 20,000 tokens, ask for word-for-word quotes before the actual task, which grounds the response in the real text [6]. And give it explicit permission to admit uncertainty, which the docs say can drastically reduce false information [6]. In Claude or ChatGPT this costs you one paragraph of instruction, and it converts an unverifiable essay into a list of claims you can spot-check in five minutes.

Ground truth comes from the environment, not the transcript

Anthropic’s guidance on agents puts it plainly: during execution it is crucial for agents to gain ground truth from the environment at each step, such as tool call results or code execution, to assess progress [7]. The word “environment” is doing real work there. A failing test is ground truth. A sentence saying the tests pass is a sentence.

So hand the agent the check as something it has to run, not as an instruction to be careful. In Claude Code or Cursor, that means the suite runs and the run is not finished until it is green. For a spreadsheet job, it means the agent recomputes the total from the rows rather than reporting the total it wrote. For a research task, it means fetching each URL it cites rather than producing URLs that look right.

This also changes what a long run is worth. The Riemann run’s first 650 ideas produced nothing usable [1], and a search like that only functions because the dead ends were detected as dead ends. In an unchecked run, a wrong step at minute 4 becomes the foundation for the next 40 minutes of work, and you get a large, internally consistent, wrong artefact. Anthropic recommends stopping conditions such as a maximum number of iterations to maintain control, along with extensive testing in sandboxed environments and appropriate guardrails [7]. Set the cap before you start, because you will not be watching.

Checking has to cost less than doing

The economics of this are simple and people get them wrong in one direction. Tokens are rarely your constraint at small scale, as the roughly $2,000 OpenAI attributes to finding all ten of those mathematics results suggests [2]. Your minutes are. If verifying an output takes as long as producing it yourself would have, the run saved you typing and nothing else, and you have added a failure mode.

The way to keep the check cheap is to make it repeatable rather than thorough. A command you run every time beats a careful read you do once and then abandon. Where a full check is genuinely expensive, sample: check the three outputs where being wrong costs the most, not a random tenth of everything. And when the check keeps passing across 20 runs, that is evidence about the task, not permission to stop checking, because the run that breaks the streak will look exactly like the others.

What still goes wrong

A check tests what it tests, and nothing else. A green test suite means the code passes those tests, which is a much narrower claim than “the code is correct”, and the gap between the two is where the expensive mistakes live. Anthropic’s own hallucination guidance ends on the same note: the techniques significantly reduce hallucinations but do not eliminate them entirely, and critical information should always be validated, especially for high-stakes decisions [6].

The mathematics example is also not a template, and it is worth being honest about why. Anthropic says directly that it does not expect the techniques Claude used to lead to proving the Riemann hypothesis [1]. More to the point for you, mathematics is unusual in having a mechanical checker at all. Most of what you do has no equivalent of Lean sitting there waiting to reject a bad answer, and the honest response to that is not to build a worse one and call it verification. It is to keep that work in group three and treat it accordingly.

Some of that work is not yours to ship on a model’s say-so regardless. Anthropic’s Usage Policy, effective 15 September 2025, requires that where its products are used for advice, recommendations or subjective decision-making that directly affects individuals, a qualified professional in that field reviews the content or decision before it is disseminated or finalised, and that anyone shown the output is told AI helped produce it [8]. The domains named include legal, healthcare, insurance, finance, employment and housing, academic testing, accreditation and admissions, and media or professional journalistic content generated automatically and published for external consumption [8]. No check you build removes that review step, and the review is a person, not a command.

Prompts from this guide

check-plan

Before you start, write the check.

Task: {task}
What I will do with the output: {downstream_use}

Answer in three parts and nothing else:

1. The check. One command, script, or comparison that would fail if the
   output were wrong. If no such check exists for this task, say so
   plainly rather than inventing one.
2. What a failure looks like. The specific output, error or difference
   I would see if the work were wrong.
3. What cannot be checked. The parts of this task where the only test
   is a person's judgement.

Do not start the task.