How to judge a new coding agent before you switch

A new terminal coding agent lands every few weeks, someone you respect posts a screenshot of it doing something impressive, and you spend the evening wondering whether the thing you already pay for is now the slow option. You are not short of choices. You are short of a way to decide between them that does not eat a week.

This guide is for the solo operator or the two-to-five person team that writes code and pays for tools out of its own revenue. If you have a security team, a procurement queue and a data processing agreement template, the work below is already someone’s job and they will do it better. What follows is the version you can run yourself: three questions that decide the answer, and a bench test that fits in an afternoon.

An agent competes on capability, price and custody, and only the first one is in the demo

Every launch post is about capability. That is the axis vendors control, and it is the axis that changes fastest, which makes it the worst basis for a decision that you will live with for a year.

Meta’s Muse Code, released in beta on 5 August 2026, is a fair example of the genre [8]. The pitch was architectural. Mark Zuckerberg described the design in the launch coverage: when a job is big enough it “fans out to separate sub-agents working in parallel in isolated worktrees” [8]. Meta’s AI chief Alexandr Wang framed the appeal partly on price, saying that “for a lot of workflows and a lot of use cases, this can be an incredibly good option, especially from a cost perspective” [8]. Both claims may be true. Neither is checkable from a blog post, and both will be matched or beaten by someone else before you finish evaluating.

The two axes the launch post will not settle for you are what the tool costs at your actual usage, and what the vendor is allowed to do with the code you feed it. Those are written down, they are stable for months at a time, and they differ between products in ways that matter more to a small business than a benchmark score.

Price the agent against a session, not against a month

Sticker prices across the main options sit in a narrow band. Claude Pro is $20 a month billed monthly, or $17 a month on an annual plan billed at $200 up front, and it includes Claude Code [1]. Claude Max starts at $100 a month for 5x or 20x more usage than Pro [1]. On the OpenAI side, ChatGPT Go is $8 a month, Plus is $20 and lists “Codex on the web, in the CLI, in the IDE extension, and on iOS”, and Pro starts at $100 a month with 5x or 20x higher rate limits than Plus [3]. Cursor has a free Hobby tier with limited agent requests and a Pro tier at $20 a month [6].

The monthly number is the least interesting part. What you are actually buying is throughput inside a window. Claude states that Pro gives at least 5x more usage per 5-hour session than Free [1]. OpenAI publishes Codex limits the same way, as a band per 5-hour period that depends on which model you point at the task: on Plus the published estimates run from 5 to 45 messages on the heaviest model up to 250 to 2,000 on the lightest, with Pro multiplying whichever band applies by 5 or 20 [3]. So the question that decides your bill is not how many hours a month you code. It is how many of your working sessions run into the ceiling, because that is the moment you either stop, wait, or upgrade.

Team pricing is where a switch gets expensive without anyone noticing. Claude Team standard seats are $20 per seat per month billed annually, or $25 monthly, and premium seats are $100 annually or $125 monthly [1]. ChatGPT Business is $20 per user per month billed annually, or $25 monthly [3]. Cursor Teams Standard is $40 per user per month [6]. Multiply by five people and a switch that looked like a personal preference becomes a line item you have to justify.

Run that with your own numbers before you read another comparison thread. For most small teams the subscription is the rounding error and the re-tuning time is the real cost, which is the opposite of how the decision usually gets discussed.

The account you sign in with decides what happens to your code

This is the axis that is easy to check and easy to skip, and it is the one where two products that look identical in a terminal window are legally different things.

Anthropic’s published policy separates consumer and commercial accounts. For Free, Pro and Max accounts, the documentation says Anthropic “will train new models using data from Free, Pro, and Max accounts when this setting is on (including when you use Claude Code from these accounts)” [2]. For commercial accounts, meaning Team, Enterprise, API and third-party platforms, the policy is that Anthropic does not train generative models on code or prompts sent to Claude Code under commercial terms unless the customer has chosen to provide data, for example through the Development Partner Program [2]. Retention follows the same split: consumer accounts with model improvement on are retained for 5 years, consumer accounts with it off for 30 days, and commercial accounts for 30 days as standard, with zero data retention available per organisation to qualified Enterprise accounts rather than being included by default [2].

OpenAI draws the line in the same place. For consumer accounts, conversations are used to improve models unless you turn off “Improve the model for everyone” in Settings under Data Controls, and the page states you can change that setting at any time with no restrictions [4]. For business products and the API, the policy page states plainly that “We do not train our models on your data by default”, and that OpenAI “may securely retain API inputs and outputs for up to 30 days to provide the services and to identify abuse” before removing them [5].

Cursor exposes the same choice as a setting rather than a plan. Its pricing page says that when Privacy Mode is enabled, “we guarantee that code data is not used for training by us or our model providers” [6].

Read those three policies together and the durable rule falls out. Nobody is hiding anything. The default simply depends on which door you came through, and the cheap door is usually the one where your code helps train the model. That is a fair trade for a hobby project and an unacceptable one for client work under a confidentiality clause, and the tool will look exactly the same either way.

A bench test that fits in one afternoon

Pick a task you were going to do anyway. Not a toy, not a greenfield demo. Something that touches at least three files in a repository you know well, has an existing test you can run, and would take you 45 minutes by hand. The point is to test the agent against work whose correct answer you can recognise.

Run the same task on the incumbent and the challenger, each in its own branch, each starting from a clean tree. Give both the same instructions, written once and pasted twice. If you improvise better instructions for the second run, you have measured your prompting, not the tool.

Then judge the output on review time rather than wall-clock time. An agent that finishes in 4 minutes and hands you an entangled diff across nine files is slower than one that takes 15 minutes and produces something you can read top to bottom. Time yourself reviewing each diff, note how many times you had to open a file to understand a change, and note whether the tests passed on the first run or after you fixed something the agent broke quietly. Those three numbers are worth more than any published benchmark, because they are measured on your codebase and your standards.

Keep the branches. In three months when the next agent launches, you already have a fixed task, a written instruction and two recorded results, and the new evaluation takes an hour instead of a day.

Your switching cost is the instructions, not the subscription

The subscription is the easy part to leave. What actually locks you in is everything you wrote around the tool: the rules file that tells it your conventions, the project instructions that stop it reaching for the wrong library, the custom commands, the MCP server configuration, the accumulated small corrections that made it usable. Rebuild that for a new agent and you lose the afternoon you costed in the calculator above, per person.

So write those instructions to be portable. Keep them as plain markdown in the repository rather than in a vendor’s web settings panel, describe your conventions in prose that any competent agent could follow rather than in one product’s syntax, and keep the vendor-specific configuration in a separate thin file. You will not achieve perfect portability. You can get most of the way there, and it converts a switch from a rewrite into a copy.

This matters because you do not always choose the moment. Google’s own notice records that from 18 June 2026, the Gemini Code Assist IDE extensions and Gemini CLI stopped serving requests for the Gemini Code Assist for individuals, Google AI Pro and Google AI Ultra tiers, with those users directed to migrate to the Antigravity family of products, while access using Gemini Code Assist Standard or Enterprise subscriptions remained unchanged [7]. Nobody using it on a personal account chose to move. The consumer tier is the one that gets withdrawn, and the plain-text instructions in your repository are the part that survives it.

What still goes wrong

The biggest failure is quiet tier drift. You evaluate on a commercial account, confirm the policy, adopt the tool, and six months later a contractor installs it and signs in with their personal subscription because that is the login they had. Nothing breaks, nothing warns you, and the retention period on that person’s sessions is now 5 years rather than 30 days [2]. The only defence is a written rule about which account is used for client repositories, and someone checking it occasionally.

The second is that these policies are current as of the dates cited here and not before or after. Vendors reorganise plans and rewrite privacy pages regularly. Anthropic’s consumer privacy settings can be changed at any time [2], and so can OpenAI’s [4], which means a policy you verified in March may be describing a different default by September, and a teammate may have flipped the toggle without telling you. Re-read the page you are relying on when you renew, not when you first buy.

The third limit is the bench test itself. One task on one repository tells you about that task on that repository. It will not tell you how the agent behaves on a code style it has never seen, on a large refactor, or at 11pm when you stop reading diffs carefully. Treat the result as evidence that the tool is worth a two-week trial, not as proof it is better. And no test protocol protects you from the underlying problem, which is that you remain responsible for code you did not write and only skimmed. The agent changes how fast the diff arrives. It does not change who signs off on it.