When an AI agent goes wrong, yours or your vendor's

The version of agent risk everyone plans for is your own agent deleting your own files. The version that lands is quieter and less symmetrical. Anthropic’s account of an operation it detected in mid-September 2025 describes a group it assesses “with high confidence was a Chinese state-sponsored group” manipulating Claude Code into “attempting infiltration into roughly thirty global targets” and succeeding in a small number of cases, with the threat actor able “to use AI to perform 80-90% of the campaign” and human intervention required “only sporadically (perhaps 4-6 critical decision points per hacking campaign)” [5]. The agent was a vendor’s product. The victims were other people’s systems. Nobody in that story had a checkbox to untick.

This guide is for a solo operator or small team that has already connected an AI tool to something real: an inbox, a repository, a payments dashboard, a folder of client files. The goal is not to make an incident impossible, which you cannot do. It is to make the damage bounded and the recovery boring. It is not for organisations with a security team, a procurement process and a lawyer on retainer, who have levers you do not. It is not legal advice either. Every contract quote below is from a public standard agreement on the date given, and the one you signed may differ.

The agent with your keys is a third party, whichever logo is on it

There are two shapes to this risk and they need different answers. The first is the one you control: an agent you deployed does something outside what you intended, on systems you own. The second is the one you do not control: a vendor’s agent, running in a context you never see, does something to somebody else, and the fallout reaches you as a suspended account, a revoked key, an outage, or a client asking whether you use that product.

Policy is moving on the second shape, slowly and upstream. On 4 August 2026 Reuters reported that staff from Meta, Anthropic, Google and OpenAI “will meet with U.S. President Donald Trump’s advisers on Tuesday about voluntary safety testing for advanced AI models”, and that the meeting “follows disclosures from OpenAI and Anthropic that their AI tools breached the systems of other companies”; the same report noted that the administration had said in June it would ask the companies developing those models to “voluntarily submit them for government tests up to 30 days before they plan to release them to the public” [8]. Read the words carefully. Voluntary, pre-release, model-level. Nothing in that process inspects the token you pasted into an integration last March, and nothing in it pays your client back.

So the useful frame is not regulatory. Treat every agent, including the one you pay for and trust, as a contractor holding your keys who does not sleep, is occasionally confused by things it reads, and cannot be un-hired retroactively. Everything below follows from that.

A permission rule is a control, an instruction is a request

The most useful sentence in any agent documentation sits in Claude Code’s permissions page: “Permission rules are enforced by Claude Code, not by the model. Instructions in your prompt or CLAUDE.md shape what Claude tries to do, but they don’t change what Claude Code allows” [6]. Every “never touch production” line you write in a system prompt is a request. Requests hold right up until something unusual arrives in the input.

The enforced layer is rules, and their order matters more than their wording. “Rules are evaluated in order: deny, then ask, then allow. The first match in that order determines the outcome, and rule specificity doesn’t change the order” [6]. Deny is where your attention pays off. A Read deny rule on a path also blocks the Edit and Write tools on that path, “including creating a new file there”, though NotebookEdit is not covered and needs its own Edit deny rule [6].

Two documented gaps are worth memorising, because their shape repeats in every agent product. Restricting web fetches by domain does not restrict the network: “using WebFetch alone doesn’t prevent network access. If Bash is allowed, Claude can still use curl, wget, or other tools to reach any URL” [6]. And file deny rules cover the agent’s own file tools and the file commands it recognises in the shell, but “they don’t apply to arbitrary subprocesses that read or write files indirectly, like a Python or Node script that opens files itself”, for which the documentation points at OS-level enforcement with the sandbox [6]. A control that sits above the layer it is protecting is a suggestion with good posture.

The mode that switches most of the system off is at least honest about it. bypassPermissions “Skips permission prompts, except for the actions no mode auto-approves”, and the docs say to “Only use this mode in isolated environments like containers or VMs where Claude Code can’t cause damage” [6]. If anyone other than you configures a machine that runs agents, permissions.disableBypassPermissionsMode can be set to "disable" in a settings file, and the documentation notes such settings are “most useful in managed settings where they can’t be overridden” [6].

Grant the smaller thing on purpose

The Model Context Protocol’s security guidance names the failure plainly under scope minimisation. A stolen broad token produces “Expanded blast radius: stolen broad token enables unrelated tool/resource access”, and “Using wildcard or omnibus scopes (*, all, full-access)” is listed among the common mistakes [7]. The recommended pattern is a “Minimal initial scope set” of low-risk read operations, with “Incremental elevation via targeted WWW-Authenticate scope="..." challenges when privileged operations are first attempted” [7].

In operator terms that means read-only wherever reading is the actual job. A separate account for the agent rather than the login that runs your business, so that revoking it does not lock you out of your own company. One folder rather than the whole drive. A test repository before the real one. A payment method with a low limit rather than the card that pays your rent. None of this is theatre. Each choice converts an incident from “everything” into “that one thing”.

The same guidance tells clients to warn that “MCP servers run with the same privileges as the client” [7]. A one-click integration inherits whatever the host application can already do, including every account you are signed into. It also forbids passing tokens through: “MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server”, partly because otherwise a downstream service’s logs “may show requests that appear to come from a different source with a different identity, rather than the MCP server that is actually forwarding the tokens”, which makes “incident investigation, controls, and auditing more difficult” [7]. If you are choosing between integrations, the one that asks for its own narrow credential is better than the one that offers to reuse a token you already have.

What the contract pays out is roughly what you paid in

Anthropic’s Commercial Terms, effective 17 June 2025, limit the liability of each party to “Fees paid by Customer for the Services in the previous 12 months” (L.3.a) [1]. OpenAI’s Services Agreement, effective 1 January 2026, has the same shape: total liability “will not exceed the total amount Customer paid to OpenAI during the twelve months immediately prior to the event giving rise to liability” (14.2) [2]. On a personal or small-team subscription that ceiling is 12 months of a small monthly number. It does not scale with what you let the agent touch, which is the only variable that decides how bad the day gets.

The indemnities run the direction you would expect once you look. Anthropic “will defend Customer” against a claim “alleging that Customer’s paid use of the Services” violates a third-party intellectual property right (K.1), while Customer “will defend Anthropic” against claims related to Customer’s inputs or “use of the Services in violation of the Usage Policy” (K.2) [1]. OpenAI’s customer indemnity covers claims related to “use of the Services in violation of this Agreement”, Customer Applications and Customer Content (13.2) [2]. Add the two clauses that do the quiet work: “It is Customer’s responsibility to evaluate whether Outputs are appropriate for Customer’s use case” (D.3), and “Customer is responsible for all activity under its account” (D.5) [1].

Read together, the allocation is unambiguous. The vendor carries model-level intellectual property risk. You carry the consequences of whatever the agent did while logged in as you. That is ordinary for software, but agents make it heavier, because the account now acts while you sleep. At this price point you are not renegotiating the cap, so your levers are the scope you grant and, if the work justifies it, insurance.

You will hear about it late, and the clock is not yours

On personal data, the vendor commitment is real but undated. OpenAI’s Data Processing Addendum, effective 1 January 2026, says “OpenAI will notify Customer without undue delay after becoming aware of any Personal Data Breach” (2.7) [3]. There is no hour count in it.

Your own clock is specific. If you handle EU personal data as a controller, Article 33(1) GDPR requires you to notify the supervisory authority “without undue delay and, where feasible, not later than 72 hours after having become aware of it”, and Article 33(2) requires that “the processor shall notify the controller without undue delay after becoming aware of a personal data breach” [4]. Your 72 hours starts when you become aware. Your vendor’s undefined delay sits inside that window, invisible to you and unbudgeted. Plan for learning late and moving fast, not for a timely phone call.

Two more items in the same addendum belong in a calendar, not in a folder. Sub-processor changes are announced by blog post, a notification within the Services, or email, and “Customer may object to the use of such additional Sub-processor within 30 days of receiving notice” (2.9) [3]. The audit right is capped at “no more than once per year”, and OpenAI may answer it with “a summary of the Audit Reports relevant to OpenAI’s compliance with this DPA” rather than an audit (2.8) [3]. If you have told a client you know every party that can touch their data, that statement stays true only if somebody reads the notices.

The obligation runs both ways, and yours is immediate. Anthropic’s terms say Customer “will promptly notify Anthropic if Customer believes the account” has been compromised (D.5) [1].

The first 48 hours are a revocation problem

When something does go wrong, the actions that matter are dull: revoke the OAuth grants, rotate the keys, pull the logs, tell the people who need telling, in that order. None of it is technically hard. Speed depends almost entirely on a decision you made months earlier, which is whether you can list what holds a credential without going looking.

If you cannot answer that in 10 minutes, day one of an incident is spent building an inventory instead of closing the hole. Build it now, on one page, one row per connected tool: what it can reach, whose account it authenticates as, where the credential lives, how to revoke it, and who has to be told if it leaks. Then price the rotation, because it is the only part of this that is arithmetic.

Run that number before you need it, then look at what it implies. 3 hours of rotation is a bad afternoon. 15 hours is a week in which you are also apologising to clients. The fix is not faster typing. It is fewer live credentials, narrower ones, and a note of where they are.

What still goes wrong

Permission systems are per product and inconsistent. The specifics quoted here are one vendor’s model, chosen because it is documented in public and unusually blunt about its own gaps [6]. Your automation tool, your code editor and your assistant each use different words for scopes, store credentials in different places, and fail in ways their documentation may not spell out. There is no single console that shows every grant an agent holds across your stack, so a hand-built inventory stays accurate only if you edit it on the day you connect something new.

Nothing in this guide helps with the case in the opening paragraph. When a vendor’s agent harms a third party, you have no control, no visibility and, under the caps quoted above, a recovery limited to what you paid that vendor in a year [1][2]. What segmentation buys you there is narrower: your data was in fewer places, so fewer of your clients are in the blast radius. The public policy work aimed at that problem is still voluntary and still upstream of the parts you touch [8].

Full segmentation of a one-person business costs real hours, and whether those hours are better spent on it than on the work that pays is a judgement only you can make. The minimum that pays for itself is smaller than it looks: know where your credentials are, keep them narrow, and never run an unattended agent with its permission checks turned off. Everything past that is a decision about how much of your business is reachable from one token.