What AI still gets wrong

The failures you meet in practice are not the ones in the headlines. Nothing dramatic happens. A quote in your newsletter turns out to belong to nobody. A margin in your pricing sheet is off by a factor of ten because one line of the arithmetic was wrong and the rest was beautiful. A client asks where a statistic came from and the link goes to a page that does not exist. Every one of these gets past you for the same reason: the output looked finished.

This guide names the seven failure classes a solo operator or small-team owner will actually hit, explains each from the way the model works rather than from folklore, and gives the cheapest check that catches it. It is written for people shipping ordinary business work, such as proposals, emails, listings, spreadsheets and code for their own site. It is not a safety review for regulated deployment. If you are automating consequential decisions about credit, hiring, housing, insurance or medical care, what you need is a risk process rather than a set of habits, and NIST publishes one for generative AI [3].

Invented facts and sources look exactly like real ones

The most common failure is a claim that is plausible, specific and false. A study that was never run. A section number in a regulation that stops one section short. A URL built out of the right domain and a guessed path. NIST has a name for it. Confabulation, in its generative AI profile, is the phenomenon in which systems “generate and confidently present erroneous or false content in response to prompts,” and the profile adds that outputs “may also include confabulated logic or citations that purport to justify or explain the system’s answer, which may further mislead humans into inappropriately trusting the system’s output” [3].

The reason sits in the mechanism. A language model produces the next likely piece of text given everything before it, and likely is not the same as true. A fake citation is highly likely text, because real citations look like that. The model is not retrieving a fact and failing. It is generating the shape of a fact. OpenAI’s own account of the problem puts the incentive plainly: “standard training and evaluation procedures reward guessing over acknowledging uncertainty,” because a model graded “only on accuracy” is “encouraged to guess rather than say ‘I don’t know’” [2].

The cheap check is to open every link and every named source yourself, not to see whether it looks right but to see whether it is there. If the model is working from documents you supplied, Anthropic’s guidance is to “make Claude’s response auditable by having it cite quotes and sources for each of its claims,” then have it check each claim afterwards by finding a supporting quote, on the rule that “if it can’t find a quote, it must retract the claim” [1]. That single instruction turns a paragraph you have to fact-check into a paragraph that fact-checks itself.

Numbers are the weakest part of any output

Text that is 95% right reads fine. A number that is 95% right is wrong. The same mechanism that produces a plausible citation produces a plausible total, which is why a sum can sit under rows it does not match, a percentage can be computed against the wrong base, and a currency conversion can use a rate that came from nowhere in particular. The arithmetic is written, not calculated, and written arithmetic reads as confidently as any other sentence.

The check is boring and it works. Never accept a number you did not see calculated. Ask for the formula and the inputs rather than the result, and put the formula in your own spreadsheet where you can see it working. If the tool can run code, make it do the calculation in code and show you the code it ran. If it cannot, recompute a sample of rows by hand, taking some from the middle and the end rather than only the first two. When a recomputed row disagrees, ask for the working rather than for a correction, and read the working yourself.

Confidence and agreement are writing styles, not measurements

Two of the signals you would normally use to judge an answer do not work here. The first is tone. The same even, professional register comes out for a fact the model has seen ten thousand times and for a detail it assembled from nothing. That is exactly NIST’s point about confabulation: false content arrives confidently presented, and confabulated logic or citations make people trust the output more rather than less [3]. The second signal is agreement. Push back on an answer and the model may come round to your view, which ruins the most natural way to test something, since the natural way to test something is to challenge it.

Both have the same root. Fluency and agreeableness are what got rewarded. Training and evaluation favour answers that score, and an answer hedged into uselessness scores badly, so the model learned to commit [2]. Agreement is a well enough known failure that vendors write rules against it. OpenAI’s Model Spec lists “Don’t be sycophantic” among its honesty rules, directly alongside “Do not lie,” and describes the assistant as something that “shouldn’t just say ‘yes’ to everything (like a sycophant)” and may “politely push back” instead [4].

The two mitigations are different. For tone, remove the penalty on uncertainty in so many words. Anthropic’s guidance is to “allow Claude to say ‘I don’t know’” and gives you the sentence to paste: “If you’re unsure about any aspect or if the report lacks necessary information, say ‘I don’t have enough information to confidently assess this’” [1]. For agreement, never verify by asking the same model whether it is sure. Check against a source, or against a second model given the original question with no hint of what the first one said. If you want to test a claim inside the conversation, state the opposite of what you believe. Agreement with that too tells you the answer was never held.

Knowledge stops at a date, and the model rarely says so

Every model is trained up to a point in time and then stops learning. Ask about a price, a policy, a tax threshold, a product tier or a version number, and you may get an answer that was accurate a year ago and is quietly stale now, delivered in the same tone as everything else. Google’s own guidance for the Gemini API states that “many AI models are experimental and may present factually inaccurate information, hallucinate, or otherwise produce problematic outputs,” and its recommended fix is grounding: connecting the model to real-time web content so it can cite verifiable sources beyond its knowledge cutoff [5].

The cheap check is to turn on the tool that searches for anything time-sensitive, and then still open the vendor’s own page. Grounding reduces staleness rather than removing it, because a search result can itself be old or wrong and the model still has to read it correctly. Treat pricing, tax thresholds, platform rules and API limits as facts you look up rather than facts you ask about. When you do ask, require the answer to state the date its information comes from, and pay attention on the occasions when it cannot.

Long tasks drift away from your instructions

On a long task, the constraint you cared about most is the one most likely to lapse. Never mention the competitor. Always use British spelling. Keep every section under 200 words. Instructions like these are given once, early, and then pushed further and further back by everything that follows them.

Position matters more here than you would expect. Anthropic’s long-context guidance is to “place your long documents and inputs near the top of your prompt, above your query, instructions, and examples,” and states that “queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs” [6]. Read that in reverse and it tells you something about drift. If material at the end carries that much extra weight, material stranded at the start carries correspondingly less.

The cheap check is to restate your rules at the end of every long prompt, as a short numbered set you can paste again unchanged. On anything running longer than a few exchanges, run a compliance pass as its own step: ask for each rule to be listed with the line in the draft that satisfies it. A rule that cannot be quoted against has already been dropped.

An agent that can act can act wrongly

The moment a model can run commands, send email or edit files, its mistakes stop being drafts. Two things go wrong. The first is ordinary error at speed, such as a delete that matches more than you meant, a message sent to the wrong list, or an edit applied across a whole folder. The second is prompt injection, which Anthropic defines as “a technique where an attacker attempts to override or manipulate an AI assistant’s instructions by inserting malicious text” [7]. That text does not have to come from you. It can sit in a web page, a PDF or a support ticket the agent reads while doing what you asked.

Vendors treat this as live rather than solved. Claude Code’s security page describes a permission architecture in which, in its manual mode, the tool starts with read-only permissions, asks before it edits files or runs commands that can modify your system, and can write only to the folder it was started in and that folder’s subfolders. Commands that fetch content from the web, “such as curl and wget,” are not auto-approved by default. The listed best practices for untrusted content are to review suggested commands before approval, avoid piping untrusted content directly to the model, verify proposed changes to critical files, and use virtual machines to run scripts and make tool calls, especially when interacting with external web services. The page also carries the sentence that matters most, which is that “no system is completely immune to all attacks” [7].

The cheap check is to give the agent the smallest blast radius that still lets it do the job. A copy of the folder rather than the folder. A test inbox rather than the real one. Read-only credentials wherever reading is enough. Keep approval prompts on for anything that deletes, sends or publishes, and read the command itself rather than the summary of the command. If the agent fetched a web page earlier in the session, be more suspicious of what it does next, not less.

Pasted material leaves your desk

The last failure is not the model’s. Everything you paste, whether a client contract, a payroll sheet, an unreleased price list or a colleague’s medical note, goes to a third party and lands under whatever terms your account happens to carry. Those terms are not the same across tiers. OpenAI’s enterprise privacy page states that “by default, we do not use your business data for training our models,” and, of the data it does use to develop models, that “we also use data from versions of ChatGPT and other services for individuals.” The same page says of its business products that “any deleted conversations are removed from our systems within 30 days, unless we are legally required to retain them” [8].

The cheap check is to ask, before pasting, whether you would email this to a supplier. If not, redact it. Names become roles, real figures become representative ones, the client becomes Client A. Redaction rarely costs you output quality, because the model needs the structure rather than the identities. Then settle the account question once, in writing: establish which tier you are on and what its published terms say about training and retention. It takes ten minutes and it stops you re-asking the question forever.

What still goes wrong

Every check here catches the failure it is aimed at and misses the rest. Requiring quotes stops invented sources but not misread ones, because a real quote can be pulled from a paragraph that says the opposite. Recomputing numbers stops arithmetic drift but not a wrong assumption fed in at the top. Redaction protects the client whose name you removed and not the one in the attachment you forgot. Anthropic says the same thing about its own advice: “while these techniques significantly reduce hallucinations, they don’t eliminate them entirely. Always validate critical information, especially for high-stakes decisions” [1].

The deeper limit is that verification takes time, and saving time was the reason you reached for the tool. That tension does not resolve, it gets managed. Verify in proportion to consequence. A draft email to a friendly client can go out lightly checked. A number in a quote, a claim in a contract, a fact with your name on it, and anything an agent will execute rather than propose all get the full pass, every time, however good the tool has become.

None of this is the right document if what you need is assurance for a regulated process. NIST’s generative AI profile is built for that job. It describes itself as “a cross-sectoral profile of and companion resource for the AI Risk Management Framework (AI RMF 1.0) for Generative AI,” offers “a set of suggested actions to help organizations govern, map, measure, and manage” the risks specific to these systems, and notes that confabulated content is especially worth monitoring “when integrating GAI into applications involving consequential decision making” [3]. If that is where you are, the document you need is longer than this one.