How to read a government model evaluation

A cheap model arrives with benchmark numbers near the frontier and a price that is not. You want to know whether it belongs in the seat where your agent runs commands, reads your inbox and touches a production key. The vendor’s benchmark table does not answer that. It was never built to.

Two government bodies have been publishing part of the answer, for free, since 2025. The UK AI Security Institute and the US Center for AI Standards and Innovation, the NIST body that serves as industry’s primary point of contact within the US government for testing commercial AI systems [3], run named models through offensive cyber and agent-security batteries and publish the results with numbers attached [1][3]. The reports are written for policymakers, which is why most operators read the headline ranking and skip the paragraph that should actually change their setup. This guide is for someone who routes work across several models and gives at least one of them tool access. If you have a security function, a vendor review process and a pentest contract, these reports are already on someone’s desk and this sits below your floor.

Two governments test named models and publish the numbers

On 16 July 2026 Moonshot AI released Kimi K3 [1], a mixture-of-experts model with 2.8 trillion total parameters, 104 billion active per token and a context window of 1,048,576 tokens, under its own Kimi K3 License [5]. Seven days later UK AISI and CAISI published a joint preliminary assessment of what it can do on cyber tasks [1][2].

The numbers are specific. On exploit development it scored 32 percent success, against 24 percent for GLM-5.2, which the institutes call the most cyber-capable open-weight model as of June 2026 [1]. On a 32-step simulated attack path called The Last Ones it reached step 17 on average, against step 11 for GLM-5.2 and 28.5 for leading US models, and completed the full chain in 1 of 10 attempts [1]. On the 41 ExploitBench tasks built around V8 engine vulnerabilities it achieved arbitrary code execution zero times, while leading US models averaged 20 of 41 [2]. The institutes’ summary is that Kimi K3 “performs significantly below the most recent frontier cyber-capable models” [1].

This is the second time the pattern has run in public. In September 2025 CAISI compared three DeepSeek models against four US models across 19 benchmarks and found that “the best U.S. model outperforms the best DeepSeek model (DeepSeek V3.1) across almost every benchmark”, with the gap largest on software engineering and cyber tasks, where the best US model evaluated solved over 20 percent more tasks [3]. It also found that one US reference model cost 35 percent less on average than the best DeepSeek model at comparable performance across 13 benchmarks [3]. Meanwhile downloads of DeepSeek models had risen nearly 1,000 percent since January 2025 [3]. A lot of people were adopting on price without checking whether the price was real.

So the first move is boring and almost nobody makes it. Before you route work to a model you have not used before, search for its name alongside AISI or CAISI. Coverage is patchy, but when a report exists it is more useful than any leaderboard, because it was produced by people with no revenue attached to the result.

The safeguard finding outranks the capability score

Skip to the safeguards paragraph first. In the Kimi K3 assessment it reads: “Kimi K3’s safeguards did not prevent it from attempting cyber exploit development or offensive cyber operations” during the evaluations [2]. That sentence has nothing to do with the ranking above it, and it is the one that touches your business.

Capability and restraint are separate axes, and a model can sit anywhere on the grid. Kimi K3 landed at lower capability with weaker refusal behaviour. The DeepSeek evaluation measured the same axis in more detail and found the spread is enormous. R1-0528 “responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models” [3]. On agent hijacking, where the malicious instruction arrives inside content the agent reads rather than from the user, DeepSeek agents were “on average, 12 times more likely than evaluated U.S. frontier models to follow malicious instructions” [3]. In simulated environments the hijacked agents sent phishing emails, downloaded and ran malware, and exfiltrated user login credentials [3].

That last finding is the one aimed at you. You are not planning to ask a model to write an exploit. The realistic failure is that your agent reads a web page, a pull request, an invoice PDF or a support email that contains instructions, and follows them. A model whose safeguards do not stop it attempting offensive work when a researcher pushes is a model whose safeguards will not stop it when a scraped page pushes. The refusal rate is not a measure of how well-behaved the model is in conversation with you. It is a measure of how much of your security depends on the model not being talked into something.

The methodology note changes what the headline means

There is one line in the Kimi K3 assessment that quietly rescales everything above it: US closed-weight models were evaluated with system-level safeguards disabled, to reduce refusals and allow measurement of maximal capabilities [2]. The comparison is therefore between one model as shipped and other models with their brakes off. It is a sound choice for the question the institutes were asking, which is what a model is capable of at the limit, and a bad basis for the question you are asking, which is what happens on a normal Tuesday with the defaults on.

Three more caveats sit in the same reports. The Kimi K3 findings “represent preliminary evaluations on a small set of public and private benchmarks” [1]. The aggregation used an approach inspired by Item Response Theory, and the confidence interval for Kimi K3 is wider than for the comparison models because its estimate came from a single benchmark [1]. And these pages get revised: the Kimi K3 assessment was published on 23 July 2026 and updated on 28 August 2026 [1], while the DeepSeek evaluation was published on 30 September 2025 and updated on 20 November 2025 [3].

The practical rule is to never carry a number out of one of these reports without the sentence next to it. “Scored 32 percent on exploit development” is a fact. “Is a third as dangerous as the frontier” is something you made up on the way to the meeting.

Sort the work by exposure, then pick the model

Model choice is the wrong first decision. Sort the work first, into three buckets, and let each bucket set its own bar.

The first bucket is text in, text out, with no side effects: drafting, summarising your own documents, classification, translation, first-pass code you will read before running. Nothing the model produces reaches a system on its own. Here the safeguard findings are close to irrelevant and you should pick on capability and price.

The second bucket reads material you did not write. Research over the open web, a model that ingests inbound email, anything summarising customer-supplied files. There are no write permissions, but there is an injection surface, and the output can steer your decisions. Prefer a model with a documented refusal record here, and read the output as a claim rather than a result.

The third bucket has hands. Tool calls, shell access, code execution, a browser session logged into something, an API key that can spend money or send mail. This is where the safeguard line decides, and where an assessment saying a model’s safeguards did not stop it attempting offensive cyber operations [2] is a straightforward reason to keep that model out of the seat, whatever it scores on coding benchmarks.

Price is what tempts people to collapse the three buckets into one, so put an actual number on the temptation. Kimi K3 runs at $2.50 per million input tokens and $14 per million output on OpenRouter [6]. Claude Opus 5 is $5 and $25, Claude Sonnet 5 is $2 and $10, and Claude Haiku 4.5 is $1 and $5 [7]. Against Opus 5 the gap is real. Against Sonnet 5 the cheaper open-weight model is more expensive on input, and batch processing at 50 percent off or cache reads at 0.1 times base input price move the comparison further [7]. Check the gap you are actually being offered before you accept any risk for it.

Run it, then ask whether that figure is worth handing shell access to a model whose refusal behaviour a government evaluation flagged. For most solo operators the honest answer is to keep the cheap model and give it only the first bucket.

The controls that hold are the ones on your side of the API

Picking a better-behaved model reduces risk. It does not remove it, and CAISI has measured how much is left.

In a red-teaming competition hosted by Gray Swan and analysed by CAISI, UK AISI and frontier AI labs, published on 23 March 2026, more than 400 participants made over 250,000 attack attempts against 13 frontier models in agentic scenarios [4]. The result: “at least one successful attack was found against all of the target frontier models” [4]. The number of successful attacks differed sharply between models, and that variation “did not correlate uniformly with model capability” [4]. Certain families of universal attacks transferred across scenarios and models, and attacks developed against more robust models were particularly likely to transfer to less robust ones, but not the other way around [4]. A stronger model is a smaller target, not a solved problem, and the attack that works on the good one works on the cheap one too.

Which leaves the permissions. In Claude Code, manual mode starts read-only and asks before it edits files, runs tests or executes commands [8]. In that mode it can write only to the folder it was started in and that folder’s subfolders [8]. The /sandbox command adds filesystem and network isolation for bash commands [8]. Commands that fetch from the web, curl and wget among them, are not auto-approved by default and can be blocked outright by adding them to the deny rules [8]. Web fetch uses a separate context window so that fetched text is not injected into the main one [8]. Anthropic reviews connectors against its listing criteria before adding them to its directory but does not security-audit or manage any MCP server, and recommends dev containers for additional isolation [8]. The sentence to keep is the plainest one on the page: “Claude Code only has the permissions you grant it. You’re responsible for reviewing proposed code and commands for safety before approval” [8]. The same page adds that “no system is completely immune to all attacks” [8].

Those controls are model-independent, which is exactly why they are worth more than the ranking. A sandbox with network access off contains a hijacked cheap model and a hijacked frontier model equally well.

What still goes wrong

Coverage is thin and arrives late. AISI and CAISI assess a handful of models, chosen because they are at the frontier or politically live, and most of the models on your router will never be evaluated by anyone with a government email address. Absence of a report is not a clean bill of health, it is absence of a report. The published ones also age in weeks rather than years: Kimi K3 was assessed 7 days after release [1], and any point release since then is a model nobody has tested.

The numbers measure a proxy, and proxies get over-read. A 32 percent success rate on exploit development [1] is not a probability that anything happens to you, and reaching step 17 of 32 on a simulated range [1] says nothing about your particular stack. What the reports establish well is relative ordering under one methodology at one moment. What they cannot establish is absolute risk in your environment, which depends far more on what you connected the model to than on the model.

Finally, cyber capability is one axis among several, and the loudest one. These evaluations say nothing about where your data is stored, how long it is kept, whose jurisdiction it sits in, or what happens to it in training, and those questions may matter more to your clients than exploit benchmarks ever will. The DeepSeek report did stray onto a second axis, finding that the models “echoed four times as many inaccurate and misleading CCP narratives as U.S. reference models did” [3], which is a useful reminder that a model can be safe to give shell access and still be the wrong thing to put in front of a customer.