Your assistant's refusals are a security surface

Somewhere in your setup there is an assistant that talks to people who are not you. A support bot on the pricing page, a custom GPT you shared with three clients, an inbox automation that drafts replies, a Claude project a contractor has access to. Every one of them says no several times a day. And when it says no, it says why, because you told it to be helpful and because every vendor’s default behaviour is to explain rather than stonewall.

That explanation is output. Output is published. Most operators review the answers their assistant gives and never read the ones it refuses to give, which means the refusals are the least-inspected text your business emits. This guide is about auditing that text before someone else does. It is for people who ship an assistant other people can talk to. If you only type into ChatGPT yourself, this is not your problem; the refusals go to you and stop there. If you run a security programme with a standing red team, you need a longer document than this one.

A refusal is an answer, and it is answering a different question

The clearest recent demonstration came from Varonis researcher Lior Adar, who probed Microsoft Copilot by collecting the reasons it gave for declining. Copilot refused, and explained itself while doing so. Adar’s summary of the method is the whole idea: “Each ‘that won’t work because…’ is an invitation to probe the ‘because.’ You don’t exploit the model. You manipulate it into cooperating” [1].

What eventually fell out was not a jailbreak. According to the writeup, “Copilot then disclosed an undocumented URL parameter — unprompted, mid-refusal — including its historical behavior and every protection put in place to disable it” [1]. It is ?autorun=1 “that enables automatic execution on page load” [1]. Varonis counts the finding, which it calls CoSnitch, as “three vulnerabilities, one click, and zero anomalous signals,” and says that “chained together, they turn a single click into a silent data-theft tool by exploiting the trust model at the heart of modern AI connectivity” [1]. Varonis says it disclosed the issue to Microsoft in December 2025 and that patches shipped on 18 August 2026 [1]. Eight months, for a finding that started with a model being conscientious about explaining its own boundaries.

The important part is that during the reconnaissance phase, nothing failed. No filter was bypassed. The model behaved exactly as designed and as recommended: refuse the bad request, tell the user why, do not be arbitrary. Every individual refusal was correct. The leak lived in the aggregate, across turns, in a transcript nobody was watching.

This is not a new category of bug wearing new clothes. It has a catalogue number. MITRE lists CWE-209, “Generation of Error Message Containing Sensitive Information,” defined as a product that “generates an error message that includes sensitive information about its environment, users, or associated data” [3]. The consequence, in MITRE’s words: “Attackers can use detailed information to refine or optimize their original attack, thereby increasing their chances of success” [3]. Suppressing verbose error output in production is the standard answer to that weakness. The LLM version is harder because the explanatory message is not a misconfiguration you can switch off. It is the product.

The system prompt was never a secret and should never have held one

Most refusal text is a paraphrase of the system prompt. When an assistant says it cannot do something, it is usually reading back, in its own words, a rule you wrote. So the audit starts one level up: what is in the instructions.

OWASP made this its own entry in the 2025 Top 10 for LLM Applications. LLM07:2025 System Prompt Leakage covers “the risk that the system prompts or instructions used to steer the behavior of the model can also contain sensitive information that was not intended to be discovered” [2]. The guidance is unusually blunt about where the fault lies: “the system prompt should not be considered a secret, nor should it be used as a security control” [2]. The leak is a symptom. The real defect is having put something in there that mattered.

The examples OWASP gives are the ones that show up in small-business deployments constantly. Internal business rules, such as “The Transaction limit is set to $5000 per day for a user” [2]. Filtering criteria, such as an instruction reading “If a user requests information about another user, always respond with ‘Sorry, I cannot assist with that request’” [2]. Permissions and roles. And credentials, which OWASP lists as “API keys, database credentials, or user tokens” [2]. Read your own system prompt with that list in hand. The discount ceiling you told the bot never to exceed, the named competitor it must not mention, the escalation threshold, the internal cost figure it uses to sanity-check a quote. None of those are secrets in the legal sense. All of them are worth money to a customer negotiating with you.

Anthropic’s own guidance points at the same fix and is honest about the cost of the alternative. Under the heading “Avoid unnecessary proprietary details,” its advice is that “If Claude doesn’t need it to perform the task, don’t include it” [4]. It also says to “Try monitoring techniques first, like output screening and post-processing, to try to catch instances of prompt leak” [4]. And it warns against the instinct most people reach for first, which is to add a paragraph telling the model to keep its instructions confidential: “Consider using leak-resistant prompt engineering strategies only when absolutely necessary. Attempts to leak-proof your prompt can add complexity that may degrade performance in other parts of the task due to increasing the complexity of the LLM’s overall task” [4]. The summary line is worth pinning above the desk: “Overly complex leak-prevention can degrade results. Balance is key” [4].

The same reasoning applies to uploaded files, which is where operators smuggle rules back in after deleting them from the prompt. OpenAI’s own builder guidance for GPTs says to “Use knowledge for reference material, not rules or behavior. Put rules, tone, and workflow guidance in instructions,” and to “Prefer positive, concrete instructions (‘Do X’) over long lists of prohibitions (‘Don’t do Y’) when possible” [8]. A long list of prohibitions is, functionally, a list of the things you are worried about, written down and handed to the model that will later be asked to explain itself. If you are building on ChatGPT specifically, note that the ground has moved: OpenAI states that “New GPT creation and publishing are not available on personal ChatGPT accounts, including Free, Go, Plus, and Pro,” and restricts creation to Business, Enterprise and Edu workspaces [8]. If your shared GPT dates from before that change, it is worth re-reading its instructions with fresh eyes.

An hour of asking why is the entire audit

The test is not clever and does not need a tool. Open your own assistant in a private window, as an outsider would see it, and spend an hour trying to get refused.

Ask for the thing you know it will decline. Then ask why. Then ask what would have made it allowed. Then ask whether a slightly different phrasing would work, whether the rule applies to a different account type, whether there is a threshold, whether there is an exception, whether it would behave differently on a different page, whether the limit is a policy or a technical block. Vary the framing rather than repeating it: come at the same boundary as a confused customer, an impatient one, a developer debugging an integration, and someone claiming to be your colleague. Twenty to thirty turns per assistant is enough to see the shape.

Then do the part that people skip. Copy the whole transcript into a fresh session with a different model and ask it to reconstruct, from the refusals alone, what rules and systems sit behind the assistant. That inversion is the finding. You are not grading whether the bot held the line; it did. You are grading what a stranger could rebuild from the way it held the line.

OpenAI’s safety guidance names this activity explicitly: “We recommend ‘red-teaming’ your application to ensure it’s robust to adversarial input” [5]. The same page is clear that instructions alone are not the defence, describing prompt engineering as something that “can help constrain the topic and tone of output text” while recommending, “Wherever possible, we recommend having a human review outputs before they are used in practice” [5]. Budget the pass as recurring work rather than a launch task, because refusal wording changes when the underlying model changes, and the underlying model changes without asking you.

Write the refusal to close the topic, not to teach the boundary

The fix is boring and it works. Refusals should be short, constant, and identical across causes. “I can’t help with that here” reveals nothing. “I can’t process refunds above the automatic approval threshold without a manager” reveals a threshold, an approval mechanism, a role, and the existence of a manual path worth attacking.

MITRE’s guidance on this predates every model you are running and still lands. It warns specifically against “inconsistent messaging that might accidentally tip off an attacker about internal state, such as whether a user account exists or not” [3]. Consistency is the actual control. If a blocked request, an unknown record, a permissions failure and an out-of-scope question all produce different sentences, you have built a lookup table, and each distinct sentence is a bit of information an attacker gets for free.

The detail you removed still has somewhere to go. Log it. The internal record can carry the rule that fired, the account, the parameter, and the reason, because that log is read by you and not by a stranger with time. What crosses the boundary to the user is a short line plus a route to a human. That split, verbose internally and terse externally, is the same discipline as suppressing stack traces in production, applied to a component that talks.

Saying less is not the same as saying nothing

There is an over-correction available here, and it fails in the opposite direction. The second Varonis case makes it visible. In RovoBlast, an attack against Atlassian’s Rovo assistant, “a single click on a link triggers the attacker’s embedded instructions and forces Rovo to accept externally supplied parameters as trusted inputs within a user’s session” [7]. The researchers found that “Rovo’s guardrails around untrusted prompts were almost non-existent,” and listed among the failures: “No warning, confirmation, and taint label indicating the session was seeded by an external parameter” [7].

So the assistant that says too much about its own machinery and the assistant that says nothing about the user’s situation are both broken. The distinction is whose state you are describing. Telling a user that this session was started by a link, that the request touches data outside their account, or that an action needs their explicit confirmation describes their circumstances and helps them refuse on your behalf. Telling them which parameter triggered the check, which rule number fired, or what would have passed describes your internals. Aim every message at the first category and none at the second.

The limits that hold are enforced outside the model

Everything above is hygiene on top of the real control, and it is worth being clear that hygiene is all it is. OWASP’s LLM06:2025 entry on Excessive Agency traces the damage to three root causes, “excessive functionality,” “excessive permissions,” and “excessive autonomy,” and its central recommendation is to “Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not” [6]. It also says to “Limit the permissions that LLM extensions are granted to other systems to the minimum necessary” and to “Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken” [6].

Read that against the Copilot case. The refusal text was the reconnaissance channel, but the reason the reconnaissance was worth doing is that Copilot could reach the victim’s data and send it outward once triggered [1]. If the token behind your assistant is scoped to one folder, one inbox label, or one read-only table, then a perfectly reconstructed map of your prompt buys an attacker a tour of very little. Varonis’s own remediation advice is a permissions argument, not a prompt argument: “Treat Copilot as a privileged insider: Apply the same access review and anomaly detection you would to a human employee with broad data access,” and “Audit connector configurations: Review which apps are connected to Copilot and whether each connection is actively necessary” [1].

Two cheap structural moves come out of OpenAI’s guidance and are worth doing before anything subtle. Constrain the input surface, because “Limiting the amount of text a user can input into the prompt helps avoid prompt injection” and “Allowing user inputs through validated dropdown fields … can be more secure than allowing open-ended text inputs” [5]. And keep a person on the consequential step, because a human reviewing outputs before they are used is still the recommendation from the vendor whose model you are running [5]. Neither is elegant. Both survive a model update, which is more than can be said for any sentence in your system prompt.

What still goes wrong

You cannot prove absence. Thirty probe questions is a sample, and a determined attacker has more patience than a Tuesday afternoon. A clean audit means you did not find a leak with the phrasings you thought of, which is genuinely worth something and is not a guarantee. The honest posture is that this pass raises the cost of reconnaissance rather than closing it, which is why the permissions work in the last section matters more than the wording work in the middle ones.

The user cost is real and nobody should pretend otherwise. Expect terse refusals to push people into the support queue. A customer who gets “I can’t help with that here” three times will conclude the bot is broken, and some of them will be right, because you will also have blocked legitimate requests you did not anticipate. The mitigation is the route to a human, and the route to a human costs you time. If your assistant exists specifically to reduce that time, you are trading part of the saving back for the security, deliberately, and the trade is worth naming out loud rather than discovering in a month.

And the audit expires. Refusal wording is a property of the model, not only of your prompt, so it changes when your vendor ships an update, when you switch providers, or when a feature you did not enable arrives enabled. The Copilot finding also sets a realistic expectation about timelines that are not in your hands: Varonis reports disclosure in December 2025 and patches shipped on 18 August 2026 [1]. When the leak is in someone else’s product, your options are the ones you control anyway, which are smaller permissions, fewer connectors, and a shorter blast radius on the day the writeup gets published.