What an LLM actually is (and isn't)

You have used one of these things for a few months. Sometimes it writes the client email in nine seconds and you feel like you hired someone. Sometimes it invents a clause in a contract you pasted, or forgets a decision you made two messages earlier, or tells you a supplier’s phone number with total composure and the number does not exist. The unpredictability is the problem. You cannot delegate to something whose failures you cannot anticipate.

The fix is not a longer prompt. It is a correct mental model of what the thing is. Once you have one, most of the strange behaviour stops being strange and starts being obvious in advance, which is the only kind of knowledge that saves you time. This guide is for someone who runs a business and wants to use these tools well. It is not for someone who wants to build with them; there is no maths here, and nothing about model architecture beyond what changes your decisions on a Tuesday.

The model predicts the next piece of text, and that is genuinely the whole trick

A large language model reads everything in front of it and produces the most plausible continuation, one small piece at a time. Those pieces are called tokens, roughly a short word or part of a word. The model does this by having been trained on an enormous quantity of text until it got very good at the guessing game. OpenAI’s own write-up on hallucination puts the mechanism plainly: arbitrary low-frequency facts, like a pet’s birthday, cannot be predicted from patterns alone, whereas spelling and parentheses follow consistent patterns, so errors there disappear with scale [5].

That is not a diminishment. Predicting text well requires holding an extraordinary amount of structure about how language, arguments, code and documents work. But it explains the shape of the tool. The model is not looking anything up. It is not reasoning over a database of facts with a true or false flag on each one. It is producing text that fits, and the same machinery that produces a well-judged rewrite of your proposal produces a well-judged-sounding invoice number that never existed. From the inside of the model, those two acts are identical.

Everything else in this guide follows from that one sentence. If you remember nothing else, remember that the output is a continuation, not a retrieval.

The context window is the entire world it can see

Whatever you paste into the conversation, plus the instructions the vendor wraps around it, plus everything it has said so far, makes up the context window. That is the model’s whole universe for that request. Anthropic is explicit about the plumbing underneath: the Messages API is stateless, which means you always send the full conversational history to the API [2]. Google now defaults the other way. Its Interactions API handles conversation state server-side and you pass the id of the previous interaction to continue, with client-side history management available as an option [3]. Either way the model holds nothing between requests. The transcript is reassembled in front of it every time, and the impression of a chat that remembers you is a product of that reassembly.

Two consequences matter to you. First, the window is a hard edge rather than a fade. If the input alone already exceeds the model’s context window, Anthropic’s API returns a 400 invalid_request_error, “prompt is too long”, on every model [1]. For conversations that run long, the documentation points to server-side compaction as the primary strategy for context management, which means older material gets condensed rather than kept [1]. Second, even well below the limit, quality degrades as the window fills. Anthropic names this directly: as token count grows, accuracy and recall degrade, a phenomenon known as context rot, which makes curating what is in context just as important as how much space is available [1].

Current windows are large. Claude Opus 5 and the other newer Claude models carry a 1M-token context window, while Claude Sonnet 4.5 and other earlier models have 200k [1]. Large is not infinite, and large is not the same as attentive. Google’s rule of thumb converts the abstraction into something you can hold: for Gemini models a token is about 4 characters, and 100 tokens is about 60 to 80 English words [4]. So a word costs a little over one token, and you can work out in advance whether your document fits.

The practical rule is to keep the window clean rather than full. A 40-page contract and a focused question will beat the same contract plus six earlier tangents about your pricing page. When you change subject, start a new conversation.

It has no notion of true or false, and it is most confident where you are least able to check

The model has no internal flag marking a statement as verified. What it has is a sense of what a correct-looking answer sounds like, which is a different thing and comes apart precisely where facts are arbitrary rather than patterned. Names, dates, prices, citations, statute numbers, phone numbers and case law are exactly the categories with no pattern to lean on, which is why they are the categories that get invented.

Worse, the training process actively rewards the guess. OpenAI’s account is that standard training and evaluation procedures reward guessing over acknowledging uncertainty [5]. On a question scored only for accuracy, guessing a birthday carries a 1 in 365 chance of a point, while saying “I don’t know” scores zero [5]. Models have been shaped by that arithmetic, away from admitting doubt. OpenAI’s proposed fix is to change the scoring: penalise confident errors more than you penalise uncertainty, and give partial credit for appropriate expressions of uncertainty [5]. Until that reform is universal, assume the tool will answer rather than decline.

This gives you a usable heuristic. The risk is not spread evenly across the output. It concentrates on the specific, checkable details, and it is invisible because the tone of a fabricated citation is identical to the tone of a real one.

It also has no access to your business unless you hand it over

There is a persistent assumption that the tool knows something about you. It does not. It cannot see your inbox, your accounting file, your CRM or last quarter’s numbers unless you paste them in or connect a tool that fetches them. When it produces a confident paragraph about your industry’s margins, it is producing plausible industry-shaped text, not your margins.

The reverse worry is more reasonable and worth getting right rather than guessing at. Whether what you paste is used to train future models is a policy question with a documented answer per vendor and per plan, not a property of the technology. Anthropic states that by default it will not use your inputs or outputs from its commercial products, naming Claude for Work, the Anthropic API and Claude Gov, to train its models, and that this changes if you explicitly report feedback or bugs, for example via the thumbs up or down button, or otherwise choose to allow it [8]. Its consumer products, Claude Free, Pro and Max, are covered by a separate policy [8]. Read the page for the specific product you are on before you paste a client list into it, and read it again when you change plans.

What it is actually good at, and what it is reliably bad at

It is good where the answer is a transformation of material you already have. Rewriting, shortening, restructuring, changing register, drafting a first version, summarising a long thread, extracting the action items from a call transcript, turning notes into a proposal, explaining a document in simpler terms, generating twenty variations so you can pick one. In all of these the source of truth is in the window, and the job is a language job. That is the machine working with the grain.

It is bad where the answer must come from outside the window and must be exactly right. Anything factual about the world that you did not supply. Anything arithmetic in prose. Anything with a legal or medical or tax consequence where a plausible answer and a correct answer look the same to you. Anything that depends on knowing your business, your contracts, or what was agreed on a call it was not given. It is also bad at knowing when it is out of its depth, for the scoring reason above [5].

There is a cost dimension worth internalising too, because it kills the fear that experimentation is expensive. Claude Opus 5 is priced at $5 per million input tokens and $25 per million output tokens [6]. GPT-5.6 Terra is $2 and $12 [7]. A million input tokens is roughly 600,000 to 800,000 English words [4]. At Opus 5 rates, a 10,000-token prompt that returns a 2,000-token answer costs about 10 cents, so running the same task three ways with better instructions costs about 30 cents [6]. The larger models cost more, and Claude Fable 5.1 runs at $10 and $50 per million [6], but the order of magnitude holds. The expensive resource is your attention on the review, not the tokens.

The three mistakes that come from a wrong mental model

The first is treating it as a search engine. You ask for a supplier, a statistic, a regulation, and take the answer as a lookup. It was a continuation. If you need a fact from the world, either give the model a source to read or use a tool with search attached and follow the link to the original. An answer without a source you have opened is not evidence.

The second is treating it as a colleague with a memory. You explain your business on Monday and assume Thursday’s conversation inherits it. Nothing is inherited unless it is in the window, either because you sent it again or because a memory feature stored it and put it back there [1][2]. The professional habit is to keep a short standing brief about your business, your customers and your tone, and paste it at the start of any conversation where it matters. Ten lines will do.

The third is treating fluency as competence. The output is graded by the same faculty that produced it, which is a sense of what good text sounds like. A weak argument in confident prose reads as a strong argument. This is where owners lose real money, by sending a proposal, a contract redline or a tax position that read beautifully and were wrong. The defence is boring and it works: decide before you read the output which specific claims you will verify, then verify those, then judge the writing.

What still goes wrong

Even with the right model in your head, some things stay unfixable. Verification takes real time, and for a long document it can take longer than writing the thing yourself, which means some tasks should never be delegated at all. You will not always know which ones in advance, and the honest answer is that you learn by getting burned on a few and keeping a list.

Model behaviour also moves. Vendors ship new versions, change defaults, change how much text a window holds and change tokenizers in ways that alter your costs. Anthropic notes that Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text, so the same prompt bills higher on a newer model than on an older one [6]. Habits you build on the specifics of one release will need re-checking. The mental model in this guide is durable because it describes the mechanism rather than a product, but the numbers in it have a shelf life, which is why every number here carries a source you can re-open.

Finally, none of this transfers accountability. If a fabricated figure reaches a client, it is your figure. The tool does not share the liability, cannot be asked what it was thinking, and has no memory of having done it.