The benchmark that separates models is the long one

On one announcement page, published on one day, the same model posts two very different scores. Grok 4.5 resolves 83.3% of Terminal Bench 2.1 and 29.0% of SWE Marathon [1]. Same model, same vendor, same set of charts. The 54-point gap is not measurement noise and it is not a caveat in a footnote. It is the distance between a test made of short, self-contained tasks and a test made of long ones, and it is the single most useful thing on the page.

Most people reading a coding-agent score are trying to answer a purchasing question: will this thing finish the work I give it. The scores answer a narrower question, which is whether the model finished the work somebody else gave it, in somebody else’s harness, at somebody else’s task length. When the two task lengths match, the number transfers. When they do not, it predicts almost nothing, and it fails in a way that looks like competence for the first several steps. This guide is for anyone choosing or routing models for coding work, in an editor like Cursor or through an API. It is not for benchmark researchers, it is not a leaderboard, and it will not tell you which model to buy.

One model, two scores, a 54-point gap

Take xAI’s Grok 4.5 page, published 16 July 2026, as a worked example, because it puts both kinds of number side by side [1]. On Terminal Bench 2.1 it reports Grok 4.5 at 83.3%, behind Fable (max) at 84.3% and GPT 5.5 (xhigh) at 83.4%, ahead of Opus 4.8 (max) and Opus 4.7 (max) at 78.9% each [1]. On SWE Marathon, which the page labels as resolution rate at pass@1, only four of those models appear: Grok 4.5 at 29.0%, Opus 4.8 (max) at 26.0%, Fable (max) at 24.0% and Opus 4.7 (max) at 16.0% [1]. Its SWE Bench Pro resolve rate is 64.7% and its DeepSWE 1.1 score is 53% [1].

Four numbers for one model, ranging from 29.0% to 83.3%. None of them is wrong. They are answers to four different questions about how long a job runs and how much has to stay consistent from the first tool call to the last. A benchmark made of one issue and one patch is a sprint. A benchmark that runs a goal across many files and many steps is a marathon. A model can be near the top of one field and mid-table in the other, and the ordering of the field can change between them, as it does here: Fable (max) leads Terminal Bench 2.1 and sits third on SWE Marathon [1].

The practical consequence is that a headline score sets your expectations for the wrong failure mode. Sprint failures are visible immediately, in the diff. Marathon failures show up at step eight, after the model has been convincing for seven, and they look like a plan quietly being replaced rather than a patch that does not apply.

A benchmark stops being useful once everyone passes it

Look again at the spread, not the ranking. Terminal Bench 2.1’s five-model field on that page covers 78.9% to 84.3%, a range of 5.4 points, and the bottom of it scores 94% of what the top scores [1]. The four models scored on SWE Marathon cover 16.0% to 29.0%, a range of 13.0 points, and the bottom scores 55% of the leader’s number [1]. The short benchmark has stopped telling models apart. The long one still does.

This is the ordinary life cycle of a benchmark, and it runs faster than most buyers expect. SWE-Bench Pro was built specifically to be hard: “The benchmark is constructed from GPL-style copyleft repositories and private proprietary codebases, creating legal and access barriers that reduce the likelihood of contamination” [4]. Its own leaderboard page still carries the sentence that justified it: “While most top models score over 70% on the verified version, the best-performing models, OpenAI GPT-5 and Claude Opus 4.1, score only 23.3% and 23.1% respectively on SWE-Bench Pro” [4]. Scroll down that same page today and the top row is Muse Spark 1.1 at 61.50±3.10, with gpt-5.4 (xHigh) at 59.10±3.56, Muse Spark at 55.00±3.60, claude-opus-4-6 (thinking) at 51.90±3.61 and gemini-3.1-pro (thinking) at 46.10±3.60 [4]. The description and the leaderboard on the same page are now roughly 38 points apart.

So treat a benchmark’s discriminating power as a perishable good. If the models you are choosing between are within a few points of each other on a test, that test has finished its job and you should stop using it to decide. Look for the evaluation where the field is still spread out, because that is the one still measuring something.

Changing the test moves the score more than changing the model

The clearest demonstration of this is a benchmark rebuild that raised scores without touching a single model. OpenAI built SWE-bench Verified after finding that “the unit tests used to evaluate the correctness of a solution are often overly specific, and in some cases are even unrelated to the issue”, along with underspecified issue descriptions and environment setup that made valid solutions fail [3]. It annotated 1,699 random samples from the SWE-bench test set, 3 annotators per sample, and “our annotation process resulted in 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues” [3]. The filtered result is the 500-sample Verified set [3]. On it, “GPT-4o resolves 33.2% of samples, more than doubling its score of 16% on the original SWE-bench” [3]. The model did not improve. The test got fairer.

The same effect shows up across versions of a live benchmark. Grok 4.5 was reported at 83.3% on Terminal Bench 2.1 in July 2026 [1]. Grok 4.6, released on 12 August 2026 with “a particular focus on long-running agents”, is reported at 26% on Terminal-Bench v3.0 on the same vendor’s site [2]. Read across those two numbers and you would conclude the newer model is dramatically worse at terminal work. You would be reading a version change as a capability change.

That is the whole trap in one sentence: benchmark identity is a name plus a version plus a harness, and vendors quote the name.

Sort your own work into sprints and marathons before you read anything

The sorting takes half an hour and it is the step that makes every score afterwards legible. Write down the last 20 things you actually handed to a coding agent. For each one, note how many files it touched, how many tool calls it plausibly needed, and whether the goal was fully specified when you sent it.

Tasks that touch one or two files, need a handful of steps, and arrive fully specified are sprints. Fix this failing test. Add a null check. Rewrite this function to use the new client. These map onto the SWE-bench family, which is built from “real GitHub issues from 12 Python repositories” and ships as a 2,294-instance full set, a human-filtered 500-instance Verified subset, a 300-instance Lite subset described as “curated for less costly evaluation”, a 480-instance Multimodal set for “issues described with visual elements”, and a 300-instance Multilingual set of “tasks from 42 repositories across 9 programming languages” [6]. If your work is mostly sprints in Python, Verified is a reasonable proxy, and the Lite set exists so you can run something similar yourself without a large bill [6].

Tasks that arrive as a sentence and expand into a plan are marathons. Migrate this schema. Make the auth module use the new token format. Get the test suite green after the dependency bump. These map onto the long-horizon evaluations, and they are also the tasks where the mismatch costs you real money, because the agent will burn a full session before you find out it went wrong. If more than about a third of your 20 are marathons, weight the long benchmark and treat the sprint score as a floor rather than a forecast.

Note what the sorting also tells you. SWE-Bench Pro is 1,865 tasks across 41 professional repositories, and its public split is the 731 instances sourced from open-source repositories under strong copyleft licences [4]. If none of those repositories resembles your stack, a good score there tells you the model is capable and tells you nothing about whether it is capable on your code.

Long-horizon numbers arrive with error bars, and you have to read them

Long benchmarks are harder to run, so they produce fewer samples, so their intervals are wide. Vendor pages report the point estimate. The interval is where the honesty is.

METR’s predeployment evaluation of GPT-5.6 Sol, published 26 June 2026, put the model’s 50% time horizon at “around 11.3hrs (95% CI: 5hrs - 40hrs)” [5]. The upper bound is roughly eight times the lower one. METR also flagged that the measurement “depends heavily on our detection and treatment of cheating attempts by the model”, and stated plainly: “we do not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol’s capabilities” [5].

That is a well-resourced independent lab declining to stand behind its own headline number. None of the three vendor announcement pages cited here attaches an interval to any benchmark figure [1][2][7], and the underlying uncertainty does not disappear because it went unmentioned. When you see a long-horizon figure quoted without an interval, assume the interval is wide, and do not build a staffing plan on the point estimate. The useful reading of “11.3 hours” is not that the agent will work for 11 hours. It is that somewhere between a morning and a working week is the region where reliability collapses, and you should find your own number inside that region.

Cost per resolved task is the comparison you can actually run

Two of the vendor pages cited here put cost next to the score rather than leaving it to a pricing page, which is a quiet admission that scores alone have stopped deciding anything. Anthropic’s Claude Opus 5 announcement, dated 24 July 2026, says that at max effort “the model performs within 0.5% of Fable 5’s peak score, but at half the cost per task” on CursorBench 3.2, and describes it as “surpassing Fable 5’s best result at just over a third of the cost” on OSWorld 2.0 [7]. It is “priced at $5 per million input tokens and $25 per million output tokens (the same as Opus 4.8)” [7]. xAI reports that “Grok 4.5 resolves tasks with 15,954 output tokens on average, about 4.2× fewer than Opus 4.8 (max) at 67,020”, at $2 per million input and $6 per million output [1]; Grok 4.6 carries the same $2 and $6 [2].

Those two facts multiply. Because Opus 4.8 bills output at the same $25 per million [7], the comparison is 15,954 tokens at $6 against 67,020 tokens at $25, which is about $0.10 of output per resolved task against about $1.68, a factor of roughly 17 before you account for whether the task gets finished. Which is why the denominator matters more than either figure. Spend divided by attempts is a bill. Spend divided by tasks that were actually resolved is a price, and a model that costs a fifth as much per attempt but resolves half as often is not cheaper.

The 20-task test nobody will run for you

No published benchmark contains your repository, and the marathon benchmarks are the ones most likely to be missing a score for the model you are considering. Building the substitute is less work than it sounds, and Anthropic’s own evaluation guidance points at the right shape: “Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals” [8]. The worked examples on that page run from 50 groups of paraphrased questions up to 1,000 tweets, depending on the task, and for multi-turn work it suggests 100 conversations graded on an ordinal scale [8]. You will not get to 100 multi-file refactors, and you do not need to.

Take 20 real tasks you have already completed, so you know the correct answer. Give each to the model cold, with no warmed-up context, and record 3 things: whether it finished, how many output tokens it spent, and where it went wrong if it did. Then grade the failures by signature rather than by severity, because the signatures are diagnostic. A model running a marathon task on sprint-shaped capability abandons the stated goal around the point where the plan stops fitting in its working attention and substitutes a plausible nearby goal. It cites a file or a function it introduced earlier in the session and never wrote. It reports a step as done without the step having produced any output. Those patterns, appearing together in the back half of long sessions, mean you are using the wrong model for the task shape, not that the prompt needs another rewrite.

Twenty tasks at a few thousand output tokens each is a rounding error against a month of agent usage, and it produces the one number no vendor can publish, which is your resolution rate on your work.

What still goes wrong

Vendor-reported numbers are self-reported. Every Grok figure quoted here comes from xAI’s own announcement pages [1][2], every Opus 5 figure from Anthropic’s [7], and neither has been reproduced by an independent evaluator in a way I could cite today. A vendor choosing which benchmarks to show is making an editorial decision, and the absence of a number is information too. When a model page reports a sprint score and no long-horizon score at all, the honest reading is that the long-horizon score was not flattering, not that it does not exist.

Your 20-task test has its own defect, which is that 20 is a small number and you will be tempted to read a 3-task difference between two models as a result. It is not. Use it to reject models that fail obviously and to catch failure signatures, not to rank two candidates that finish within a couple of tasks of each other. If the two are close, pick on cost per resolved task and switch later, which is cheap in an editor and moderately annoying in production code.

Finally, everything here has a short shelf life. SWE-Bench Pro’s own page carries a description written when 23.3% was the frontier and a leaderboard where the top row reads 61.50% [4]. Terminal-Bench’s version number moved from 2.1 to 3.0 between two model releases four weeks apart [1][2]. Every figure in this guide was read off a primary source on 5 September 2026 and several of them will be stale by the time you need them. The sorting method survives the numbers; the numbers do not.