Running the model on hardware you own

Every few months a lab publishes an open-weight model with a headline that says it runs on one consumer GPU, and a lot of operators start pricing out a machine. Meta’s Muse Glimmer, a 30-billion-parameter model announced on 10 August 2026 under the Apache 2.0 licence and described by its own release post as built “for local agentic use cases”, is the version most people have seen [1]. The instinct behind the interest is sound. An agent that runs on a box under your desk has no per-token bill, no rate limit, no network dependency, and no vendor able to change the terms halfway through your quarter.

The instinct skips a step, which is checking what “runs on one GPU” was measured on and what your job actually costs today. This guide is for a solo operator or a small team already paying for hosted models who wants to know whether owning the hardware is cheaper, more private or more durable. It is not for anyone who needs the strongest model available on their hardest problems, and it is not for teams whose compliance rules have already made this decision for them. Every specification and price below was read from the vendor’s own page on 4 September 2026.

What you are buying is predictability, not capability

Three things change when the weights sit on your disk. The marginal cost of a token goes to roughly the cost of electricity. The model keeps working when the connection does not. And nobody can reprice, restrict or retire it underneath you.

That third one is the underrated one, because hosted pricing is not a fixed fact. Google’s own pricing page lists Gemini 3.8 Flash at $0.75 per million input tokens and $3.75 per million output tokens through 31 December 2026, then $1.50 and $7.50 starting 1 January 2027 [6]. That is a doubling, published in advance, on the vendor’s page, for a model many small teams have already built around. Nothing improper is happening. It is just a reminder that your unit cost is a line in somebody else’s plan.

A locally hosted model under a permissive licence removes that particular exposure. The Apache 2.0 licence grants “a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license”, and a patent licence on the same terms, “irrevocable (except as stated in this section)” [7]. Glimmer ships under it [1], as do gpt-oss-20b [2] and Qwen3-Coder-30B-A3B-Instruct [3]. Once the file is on your disk, the terms you downloaded it under are the terms you keep.

What does not change is capability. Nothing about downloading a model makes it better than the hosted one you were using, and the model a lab gives away is generally not the model it sells.

The published numbers were measured on hardware you do not have

This is the part that decides most of these projects, and it is usually skimmed.

Meta’s own release post gives practical hardware minimums for Glimmer at bf16 precision on Hopper-class GPUs. Inference and evaluation: one 80GB H100. LoRA fine-tuning: one 80GB H100 with microbatch 1 and checkpointing. Full fine-tuning: eight 80GB H100s with FSDP or ZeRO-3 [1]. The benchmark table in the same post, with an MCP Atlas score of 75.5 and SWE-Bench Pro of 51.2 [1], sits above those requirements, not beside a consumer desktop.

Meanwhile the current consumer flagship, NVIDIA’s GeForce RTX 5090, carries 32GB of GDDR7 [5]. Eighty is more than thirty-two. The gap is closed by quantization, and Meta closed it deliberately: the release shipped with day-zero llama.cpp support and calibrated quantized builds in a dedicated repository [1].

Quantization is legitimate engineering, not a trick. It lowers the memory needed to load and run a model “by storing the weights in a lower precision while trying to preserve as much accuracy as possible”, with integer representations “like int8 or int4”, and some listed methods going lower still, to 2 bits or 1 [4]. Read that phrase slowly. Trying to preserve is not preserving. The published score belongs to the full-precision model on server hardware. The thing you will actually run is a compressed copy of it, and how much of the score survives compression is a question about your workload that only your workload can answer.

Memory goes on context, not just on weights

The second sizing mistake is treating the weight file as the whole budget. Weights are the fixed cost. Context is the variable one, and agent work is the context-hungry kind: a long file, a tool call, the log it returns, a failure, a stack trace, a retry, all accumulating in the same window.

The model cards are unusually honest about this if you read past the parameter count. OpenAI’s gpt-oss-20b holds 21B parameters with 3.6B active and is designed to run within 16GB of memory, using MXFP4 quantization on the mixture-of-experts weights [2]. Qwen3-Coder-30B-A3B-Instruct holds 30.5B parameters with 3.3B active and a native context of 262,144 tokens [3]. Its own card, under troubleshooting, tells you that if you hit out-of-memory errors you should reduce the context length to something shorter, such as 32,768 [3]. The Glimmer release post’s own sample configuration sets a context window of 32,768 to begin with [1].

The parameter counts also tell you two different things at once, and it is worth keeping them apart. The total count is roughly what has to be held in memory. The active count is roughly what runs on each token, so it drives speed. A mixture-of-experts model like gpt-oss-20b, with 3.6B of 21B active, or Qwen3-Coder-30B-A3B, with 3.3B of 30.5B active, is fast for its size but still has to hold the whole thing somewhere [2][3]. Reading only the small number leads you to buy a card that runs the model quickly right up until it will not load.

So the practical question is not “does the model fit”. It is “does the model fit at the context length my actual jobs run at, on the card I am about to buy”. A model that loads comfortably in a demo can fall over on the tenth turn of a real agent run, when the window is full of tool output. Test at your real lengths, not at a prompt.

Do the break-even before you buy anything

The comparison people run in their heads is the purchase price of a machine against a year of hosted spend they have never actually measured. Run it properly instead, with three numbers.

The first is what you spend now. Pull the last three months of invoices from your assistant subscriptions and API accounts and take the average. Not an estimate, the statement. Most small teams overestimate this figure badly, and if the real number is small, everything downstream is decided.

The second is capital: the machine, at the price you have actually been quoted, for a card with enough memory at your working context length. The third is the recurring cost of running it. The RTX 5090 lists a total graphics power of 575W [5]. That is the card under load rather than idling, and generation is the under-load part. Electricity at that draw is not enormous, but it is not zero, and it goes on the same side of the ledger as the hosted bill.

Then add the cost nobody puts in the model: your hours. Somebody installs the runner, keeps the driver current, re-downloads and re-quantizes when a new build lands, and works out why throughput halved after an update. If that somebody is you, and your hour has a price you would charge a client, the machine is more expensive than the invoice says.

One more framing helps, because the decision is rarely all or nothing. Split your work into the band that a competent mid-sized model handles with obvious failure modes, and the band where you need the best judgment you can buy. The first band is what a local machine is for, and it is usually most of the volume. The second stays hosted, on a subscription you keep. Priced that way, the machine has to beat only the portion of your bill it can actually absorb, which is a smaller and much more honest target than the whole invoice.

Privacy has two meanings and only one of them needs local hardware

“I want to run it locally for privacy” usually turns out to be one of two different requirements, and they have different answers.

The first is “I do not want my inputs training somebody’s model”. You can generally have that on hosted infrastructure, and you may already have it. OpenAI’s policy states that data submitted through the API platform after 1 March 2023 is not used to train its models unless you explicitly opt in, that inputs and outputs may be retained for up to 30 days to provide the service and identify abuse, and that zero data retention is available on eligible endpoints for qualifying use cases [8]. Read the policy attached to the specific plan you are on before you spend money to solve a problem you may not have.

The second is “this data must not leave our network”. That is a different requirement, and hardware you own is the only thing that satisfies it. If a client contract, a regulator or a court order is what is driving this, the cost comparison is secondary and you should stop optimising it. Buy the machine, and treat any money it saves as a rebate rather than the reason.

The distinction matters because the two requirements justify very different budgets. One is a preference worth a few hundred dollars of convenience. The other is a condition of doing the work at all.

Owning the weights means nobody else fixes them

The same licence that makes local models durable also tells you what you are giving up. Apache 2.0 provides the work “on an ‘AS IS’ BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied” [7]. There is no support contract, no status page that means anything to you, and no account manager to escalate to when output quality drops after you change a setting you do not remember changing.

That trade is fine when the work is the wide middle band of agent tasks with predictable failure modes: routine code edits, file organisation, scheduling, drafting against a template, multi-step jobs where a wrong answer is obvious and cheap to catch. It is a bad trade for work where being quietly wrong is expensive and you would want somebody to call.

Check the licence on the specific release rather than assuming, too. “Open weights” is a description of availability, not a standard set of terms, and what a given release grants and reserves lives in its licence file. Read the one attached to the model you are downloading, and keep a copy of it with the weights.

What still goes wrong

The failure mode of a quantized local model is quiet. It does not refuse or crash. It performs fine on the short tasks you use to spot-check it, and then drifts on the tenth tool call in a long run, in ways that look like an ordinary mistake rather than a capability ceiling. Comparing a handful of prompts against your hosted model will not surface that. Comparing error rates across a few dozen real jobs might.

The machine also becomes a pet. Drivers, runner upgrades, a new build worth re-quantizing, a fan that gets loud, an operating system update that breaks GPU discovery. None of it is hard, and all of it lands on the person who least wants it, usually in the week you are busiest. Hosted models have the opposite property: they change under you without asking, but you never maintain them.

And the arithmetic can invert. Break-even math assumes today’s hosted prices hold, and they move in both directions; the Gemini increase scheduled for 1 January 2027 is published, but cheaper tiers have arrived before and will again [6]. A machine bought to escape a bill can be sitting idle a year later next to a hosted model that got cheaper and better. That is not an argument against buying one. It is an argument for buying it because of a requirement you can name, rather than because of a headline about one GPU.