When to run an AI model on your own hardware

Every few months a headline says a model that used to need a rack of GPUs now runs on a phone. In July 2026 it was PrismML’s Bonsai 27B, a compressed build of Qwen3.6 27B that the company ships at a 3.9 GB memory footprint, runs on an iPhone 17 Pro at roughly 11 tokens per second, and releases under an Apache 2.0 license [1]. The release is real. The weights are public, the file sizes are what the company says they are, and the model card publishes its own numbers rather than hiding them [2]. None of that tells you whether you should stop paying for an API.

That is the decision sitting underneath every announcement of this kind, and it does not change when the model names do. This guide is for someone with a specific constraint: a client contract that forbids sending their documents to a third party, a job that has to run without a network, or a call volume large enough that per-token pricing became a line item you notice. It is not for someone who wants better answers than Claude, ChatGPT or Gemini give them today. Running a model on your own machine does not do that, and anyone implying otherwise is selling hardware.

Local is a distribution decision, not a quality upgrade

There are three honest reasons to run a model on hardware you control. The data is not permitted to leave your machine. The work has to happen with no network. Or your volume is high enough that the per-token bill beats the cost of the hardware. Everything else people cite tends to collapse into one of those three, or into a preference that a contract already covers.

Test the privacy reason before you act on it. OpenAI states that by default it does not use business data to train its models, that API inputs and outputs are retained for up to 30 days to provide the service and identify abuse, and that zero data retention is available for eligible endpoints on qualifying use cases [8]. If your concern is that your work might become training data, that concern is already answered in writing by the vendors, and you can put the answer in front of a client. If your concern is that a specific file is not legally allowed to be transmitted to a third party at all, and you may have to demonstrate that it was not, no contract term fixes that and local inference does. Those are different problems with different solutions, and most people who say the first one actually mean it.

The cost reason is the weakest of the three for one person. At list prices, Claude Sonnet 5 runs 2 dollars per million input tokens and 10 dollars per million output tokens, and Claude Haiku 4.5 runs 1 dollar and 5 dollars [7]. A solo operator working the model hard most of the day is still spending less per month than a single decent GPU costs, and the GPU does not come with a support line. Cost becomes a real argument at company scale, or when a single automated workflow runs thousands of times a day without a human waiting on it. Run the arithmetic below before you assume you are in that group.

What compression costs you, and where it lands

The interesting engineering in a release like Bonsai is not that a model got smaller. It is how the quality loss distributes. PrismML compresses a 27.8-billion-parameter model from roughly 54 GB at full precision down to 3.9 GB in its 1-bit form and 5.9 GB in its 1.58-bit ternary form, about 14 times and 9 times smaller respectively [1]. The headline in the announcement is that the ternary build keeps more than 95 percent of full-precision benchmark performance and the 1-bit build more than 90 percent [1].

The model card is more useful than the headline, because it breaks the average apart. Across 15 benchmarks in thinking mode, the 1-bit build scores an overall average of 76.11 against 85.07 for the full-precision baseline, which is 89.5 percent. Underneath that: math 91.66, coding 81.88, knowledge and reasoning 73.39. Then the two categories that decide whether a model can be automated: instruction-following drops to 65.74 against a baseline of 78.47, and agentic tool calling to 66.03 against 80.00. PrismML says plainly that the remaining gap concentrates in instruction-following and agentic tasks, and that agentic coding workflows were not a focus of this release [2].

That breakdown is the whole lesson, and it survives every future model. An aggregate retention figure is a portfolio return. If the work you need done is arithmetic or short code generation, a 10 percent average loss may cost you almost nothing. If the work is following a six-step instruction reliably, or driving a tool in a loop where one malformed call breaks the chain, the loss lands exactly on you and “90 percent retained” describes a model that fails your task most times you run it. Read the per-benchmark table. When a vendor does not publish one, assume the average is hiding something and go find out yourself.

The hardware number that matters is memory, and it moves with context

A resident memory footprint is a headline about an empty conversation. Bonsai’s 1-bit build needs 3.9 GB for the language model alone, but peak usage climbs to 11.6 to 12.2 GB at a 100,000-token context without key-value cache compression, and sits around 9.4 GB at a 262,000-token context with a 4-bit cache enabled [2]. So the same model is a phone-sized program when you ask it a question and a laptop-sized program when you paste in a contract.

This is why the honest hardware question is not “can I run it” but “can I run it on my longest real input”. LM Studio, a desktop app for running local models, recommends 16 GB of RAM as the baseline across platforms, at least 4 GB of dedicated video memory on Windows, an AVX2-capable processor on x64, macOS 14.0 or newer on Apple Silicon, or Ubuntu 20.04 or newer on Linux [6]. Those are minimums for the software to work, not for your workload to be comfortable.

Model file sizes give you the other half of the picture. Through Ollama, Google’s Gemma 4 family ships at 7.2 GB for the E2B build and 9.6 GB for E4B, both with a 128,000-token context, and at 7.6 GB for the 12B, 19 GB for the 26B mixture-of-experts build and 20 GB for the 31B, all with a 256,000-token context [5]. Add your operating system, your browser and whatever else is open, then add the context overhead, and a 16 GB machine stops being generous quickly.

Tokens per second decides whether anyone actually uses the thing

Speed is where local inference quietly loses people. The same Bonsai 1-bit build runs at roughly 11 tokens per second on an iPhone 17 Pro, roughly 44 on an M5 Pro laptop, and 104.8 on an H100 [1][2]. Eleven tokens per second is readable, in the sense that text appears at about the pace of someone typing quickly. It is not usable for a 900-word summary you wanted in the next few seconds, and it is painful in any loop where the model has to produce output, receive a tool result and produce more.

Two things about published throughput numbers are worth carrying forward. First, they are measured on a cold, short generation, and PrismML says directly that phone throughput is thermally limited during sustained generation [2]. A number from a 30-second demo does not describe minute six on a warm device running on battery. Second, tokens per second is not the metric you care about. Time from your keystroke to a finished, correct answer is, and that includes the model loading, the prompt being processed, and the reruns you do when the first output ignores half your instructions. Measure that end to end, on the machine that will actually run it.

The model already on your device is a different kind of tool

The most likely local model in your life is not a compressed 27B build you downloaded. It is the small one your operating system already ships. Apple’s on-device foundation model is roughly 3 billion parameters, quantized to 2 bits per weight for decoder weights with quantization-aware training, 4-bit embeddings and an 8-bit key-value cache, and trained to handle sequences up to 65,000 tokens [3].

Apple is unusually direct about what that buys. The company states the model is not designed to be a chatbot for general world knowledge [3], and the developer documentation says on-device models excel at text generation tasks like summarization, entity extraction, text and image understanding, refinement and generating creative content, then adds that when you need more reasoning capability and context size you should use Private Cloud Compute or a server model provider [4]. Using it requires a device that supports Apple Intelligence [4].

Take the shape of that advice rather than the brand. The vendor closest to the silicon is telling you the same thing the benchmark breakdown told you: small local models are good at bounded, well-shaped jobs on text you hand them, and they are not where you send open-ended reasoning. Pull the names out of the sentence and it stays true in two years.

A one-afternoon test that beats any benchmark table

Collect 20 prompts from work you have already done, including the longest input you realistically handle. Run them against whatever cloud model you use now and keep the outputs, because that is your baseline and no published number can substitute for it. Then run the same 20 against the local candidate and grade both on a rubric you write before you look at either set: did it follow every instruction, did it invent anything, would you have sent this to the client without editing.

While that runs, watch two dials. Watch peak memory at your longest input, not the resident footprint from the announcement, because that is the number that decides whether the model coexists with the rest of your machine. And time the full round trip on your own hardware, warm, after several requests in a row, not on the first cold generation. If the local model wins on your rubric, you have found a genuine result that no vendor benchmark could have given you. If it loses in a way that touches the tasks you actually care about, you have spent an afternoon instead of a quarter.

What still goes wrong

Almost every number that makes a local model sound viable comes from the company that made it. PrismML’s benchmark suite, its retention percentages and its throughput figures are all self-reported [1][2], and to the company’s credit the model card discloses where the losses land rather than stopping at the average. That is better than most, and it is still the vendor grading its own homework. Treat any retention figure as a hypothesis about your work until you have run your own 20 prompts, and be aware that a compression method tuned to score well on public benchmarks can behave differently on your unusual, specific, badly formatted real input.

The second cost is ongoing and nobody puts it in the announcement. When you run the model, you are the operations team. You handle the disk space, the driver that breaks after an update, the day the model will not load, and the decision about when to migrate to a newer build. Cloud models improve underneath you without your involvement; a local copy is frozen at the moment you downloaded it, which is a feature if you need reproducibility and a slow tax if you do not. Budget hours, not just gigabytes.

Finally, be honest about who this is not for. If you have no rule stopping you from using a hosted API, no requirement to work offline, and a bill in the low tens of dollars a month, running models locally is a hobby with a cost, and a reasonable one, but it is not an efficiency. The case for local inference is made by a constraint you can state in a sentence. If you cannot state it, you do not have one yet.