How to read an AI automation benchmark

Somebody sends you a chart. An AI agent now completes 15.8% of real freelance projects to a standard human reviewers judge at least as good as the professional’s own delivered work, up from 2.5% at the benchmark’s release under eight months earlier [1][2]. The number is real, it comes from careful people, and within a day it is being used in a conversation about whether you still need to hire. The trouble starts when you try to work out what it would mean for your Tuesday, because the number was not measured on your Tuesday. It was measured on 240 self-contained projects sourced from 358 verified Upwork freelancers, run inside a specific piece of software, under a fixed per-project spending cap, and graded by human reviewers against the gold-standard deliverable a paid professional produced [1][2].

This guide is about reading numbers like that without either dismissing them or over-crediting them. It is for a solo operator or small-team owner deciding what to delegate to a model, what to keep, and when a decision made six months ago has expired. It is not a methodology course, and it will not help you defend a procurement decision to a regulator or a board that wants formal evidence. What it will do is let you tell, in about two minutes, what a given score can and cannot tell you about your own work.

A score belongs to a model, a scaffold and a budget, not to the model alone

The Remote Labor Index publishes an automation rate for each agent it tests: the percentage of projects where the AI’s deliverable is judged at least as good as the human standard [1]. Today the board reads Fable 5 at 15.80%, Opus 4.8 at 8.33%, Codex GPT 5.5 at 6.25%, Gemini 3.7 Flash at 5.00%, and Claude Opus 4.6 at 4.17% [1]. Those look like properties of models. They are not.

Look at what produced them. Claude models ran inside Claude Code and OpenAI models inside Codex CLI, both modified to add a native computer-use tool, on a full Linux desktop VM stocked with over 30 professional applications, with an NVIDIA A100 available when a task needed one [2]. Each project got up to 24 hours of wall-clock time. The per-project budget was $50 by default and $150 for Fable 5, whose higher per-token pricing requires a larger dollar budget for most projects to finish without hitting the cap [2]. The harness also runs a worker-critic loop, in which an independent critic agent reviews each deliverable the way a demanding client would, opening files and checking the work against the brief, and the worker revises until the critic is satisfied or the budget is reached [2]. Change the harness, the budget or the loop and the number changes. The previous published leader on this same board scored 4.17% [2], and nothing in that pair of numbers separates model progress from harness progress.

So the first question to ask about any automation score is which triple it belongs to: which model, inside which scaffold, at what budget. Vendors rarely lie about this. They also rarely put it in the headline. When the triple is not stated, the score is a marketing number and should be read as one.

The budget detail is worth holding onto for a second longer, because it is the part that translates directly into your world. The average RLI project is worth $632.60 and takes a human a mean of 28.9 hours [1]. Spending up to $150 of compute to attempt one of those is a sane trade only if a person can tell quickly whether the result is any good. That is the real economics of automation at this level, and it is a fact about your review capacity rather than about the model.

Who graded the deliverable decides the number

The second question is who judged. On the Remote Labor Index, human evaluators score each deliverable on a 3-point scale, where 1 fails, 2 meets the standard and 3 exceeds it, against a gold-standard deliverable produced by a paid professional, with 94.4% inter-annotator agreement from majority voting among three independent evaluators [1][2]. That is expensive and slow, and the team kept doing it for a reason. When they checked automated AI judges against their humans, the automated judges overestimated the newer models by roughly 3 times for GPT 5.5 and about 2.5 times for Opus 4.8 [2]. Rankings survived; absolute levels did not. Their explanation is the useful part: evaluating an RLI deliverable is itself a demanding, agentic task [2]. OpenAI reports the same shape on GDPval, where experienced professionals from the same occupations grade blind without knowing which output is human, and the experimental automated grader is described as not yet as reliable as expert graders [3].

Now the failure in the other direction, which is less discussed and just as large. When OpenAI had annotators go through the original SWE-bench, they flagged 38.3% of samples for underspecified problem statements and 61.1% for unit tests that may unfairly mark valid solutions as incorrect; 68.3% of samples were filtered out of the resulting SWE-bench Verified set. GPT-4o’s score went from 16% to 33.2% on the cleaned set [6]. The model did not change. The grader did.

Put those together and you get a rule with teeth. A model-graded score is probably too high. A score graded by brittle automated tests is probably too low. Neither is fraud, and both move the headline by a factor of two or three, which is larger than most of the year-over-year progress people are arguing about.

The trend transfers, the level does not

The direction and speed of these numbers is the part you can trust and use. RLI’s frontier went from a maximum of 2.5% at release to 15.8%, more than quadrupling in under eight months [2]. METR’s task-completion time horizon, which measures the length of task a model finishes with 50% reliability, has a post-2023 doubling time of 131 days on its current dataset, against 165 days on the previous one [5]. Whatever the levels mean, a slope like that means any evaluation you did of a task category last year is now stale.

The level is much softer than it looks. When METR revised its task set for Time Horizon 1.1, growing the suite from 170 to 228 tasks, the measured 50% horizon for Claude Opus 4.5 moved from 289 minutes to 320, and for GPT-5 from 138 minutes to 214, a 55% jump [5]. Same models, same lab, different task composition. The published interval for Opus 4.5 is 170 to 729 minutes [5], and METR’s own note on limitations says error bars have historically been a factor of about 2 in each direction, adding that one of its researchers “really ha[s] no idea whether Claude’s ‘true’ time horizon is 3.5h or 6.5h” [4].

Averages hide the thing you actually need. METR reports that measured time horizons are fairly similar for math but 40 to 100 times lower for visual computer-use tasks [4]. An aggregate score across that spread is not a number about any real job. If a benchmark publishes a per-domain breakdown, read only the rows near your work and discard the headline average, because you do not delegate an average.

A 50% success rate is not a delegation threshold

There is a specific misreading that costs money, and METR names it directly: “A 50% time horizon of X hours does not mean we can delegate tasks under X hours to AIs.” Some reliability-critical and poorly verifiable tasks require success probabilities above 98% to be worth automating at all [4]. The same note points out that the metric does not settle independence: it measures the amount of serial human labour a model can replace with a 50% success rate, and METR has no tasks involving multi-turn interaction with a human, so the question of whether a model could do a person’s job unsupervised has no answer in the data [4]. Half the time, someone still does the work.

Read the automation rate the same way. An agent at 15.8% means roughly five projects in six are not judged at least as good as what the freelancer delivered [1]. That is remarkable progress and a terrible record for an unsupervised worker, at the same time, with no contradiction. The operational conclusion is not that agents can do a sixth of your work. It is that agents now produce acceptable work often enough that your review capacity, your brief quality and your handoff design have become the binding constraints. Failures are cheap when a person catches them at review and expensive when nobody looks.

Benchmark tasks are cleaner than your work

Every serious benchmark of this kind buys or commissions well-specified tasks, because a task nobody can grade is useless. Each of RLI’s 240 projects is a self-contained unit sourced directly from experienced freelance professionals across 23 Upwork domains [1], which means a stranger had to be able to complete it from the written brief alone. GDPval’s 1,320 tasks across 44 occupations in the top 9 industries were written by professionals averaging 14 years of experience, and each task received an average of 5 rounds of expert review [3].

Your backlog does not look like that. Most internal work is specified in tribal knowledge, half of it lives in your head, and the acceptance criteria are discovered during the revision. GDPval is explicit that its current version is one-shot, so it does not capture cases where a model would need to build context or improve through multiple drafts, or where real-world ambiguity would send a professional back to the client before proceeding [3]. METR makes the matching point about its own suite: all of its tasks are self-contained, whereas most long human tasks require collaboration, and it has none involving multi-turn interaction with a human [4]. It adds a sharper warning. Anything automatically gradable can be used as a training environment, and models are extensively trained that way, so building gradable tasks that do not overestimate real-world performance is genuinely hard [4].

One more finding should kill a common instinct. On SWE-Lancer, OpenAI observed that a task’s monetary value, which should be a decent proxy for engineer-hours, does not correlate with a model’s success rate [4]. Your sense of which jobs are hard is priced in human terms, and that pricing does not predict where a model fails. Which is the whole argument for testing rather than reasoning about it.

The only automation number that should move your decisions is one you measured

A private benchmark that beats every public one is smaller than you think. Take five real tasks you would plausibly delegate: recent, finished, with the brief exactly as you originally gave it and the deliverable you actually accepted. Do not improve the brief. Run each through the tooling you would really deploy, name the model and the settings in your notes, and then grade the way the careful benchmarks grade. Ask whether you would have paid for the result as delivered, without asking for changes. Score each on the same 3-point scale RLI uses: 1 fails, 2 meets the standard, 3 exceeds it [1].

Anthropic’s evaluation guidance says the same thing in engineering terms: design evals that mirror your real-world task distribution, factor in edge cases, and include ambiguous test cases where even humans would find it hard to reach a consensus [7]. That guidance also recommends prioritising volume over quality, on the grounds that more questions with slightly lower signal automated grading beats fewer questions with hand-graded ones [7]. That is right for a team shipping a product against hundreds of test cases and wrong for you at five. At five tasks you are the judge, and human judgement is precisely what both RLI and GDPval fell back on when their automated graders drifted [2][3].

The cost is trivial next to the decision it informs. Current Anthropic list prices are $10 per million input tokens and $50 per million output for Claude Fable 5.1, $5 and $25 for Claude Opus 5, and $2 and $10 for Claude Sonnet 5 [8]. Five tasks is an afternoon and a small bill. Re-run it when the frontier visibly moves, which on the current slope means every few months [2][5], and keep the old results so you can see your own trend line rather than borrowing someone else’s.

What still goes wrong

Five tasks is not statistics, and it never becomes statistics. It gives you a feel for where the line sits in your work this quarter with your briefs, which is more decision-relevant than a public number and far less defensible if someone challenges it. If you need evidence that survives scrutiny from a client, an insurer or a regulator, you need a real evaluation process and this guide is the wrong document.

The public numbers have a deeper problem you cannot fix from the outside. Most of what moves a score is invisible in the score: the harness version, the budget, the judge, the exact task mix. METR’s own revision moved one model’s measured horizon by 55% without touching the model [5], and OpenAI’s audit of SWE-bench roughly doubled a score by removing broken and underspecified tasks [6]. When a result’s scaffold is not described in enough detail for you to rebuild it, you cannot reproduce the figure even if you want to, and the honest way to read it is as an upper bound achieved under favourable conditions by people who knew the test.

And your own benchmark ages faster than you will remember to re-run it. It is a snapshot of one model version, one set of settings, and your briefing habits on the day you took it. When any of those change, the number is a historical artefact. Date it, keep it, and distrust it after a quarter.