Open-weight AI hackers now match restricted models at a fraction of the cost

A freely downloadable open-weight model, Deepseek V4.1 Flash 1017, completed 72.8 percent of objectives in a simulated corporate network intrusion — just eight points behind OpenAI's restricted GPT-6 Sol, which scored 80.8 percent in the same test. That's the headline finding from Vals AI's CyberOffense benchmark, and it lands at a moment when regulators and labs are still arguing about whether open model weights meaningfully raise security risks. The short version: the answer is increasingly yes, and the price difference makes it worse.

The benchmark matters because of how it's built. Vals simulates realistic corporate networks and drops each AI agent onto an isolated attack machine. The agent starts with nothing but a public web address and gets zero human help. Everything after that — reconnaissance, exploitation, movement through the network — has to come from the model itself. That's a much stricter test than most lab evaluations, and it explains why the results carry weight with security researchers.

How big is the gap between open and closed models?

In raw capability, it's now measured in single-digit percentage points. GPT-6 Sol, part of OpenAI's restricted Daybreak cyber program, leads the field at 80.8 percent of objectives. Deepseek V4.1 Flash 1017 sits at 72.8 percent — but a single run costs about $3, versus roughly $22 for the closed model. That's roughly a seventh of the price for most of the capability.

The harder Lighthouse test sharpens the picture. There, Daybreak took over admin control of the first simulated company, stole passwords, and used them to break into the second company. It completed 27 of 31 objectives in under three hours. Deepseek managed 21 objectives at a tenth of the cost — and, importantly, Vals found it wasn't limited solely by safety guardrails. The open model simply had less raw capability, not a leash holding it back.

The full scoreboard across both scenarios:

Model	Nova (of 21)	Lighthouse (of 31)
GPT-6 Sol Daybreak	15.7 (avg of 3 runs, range 12–19)	27
Deepseek V4.1 Flash 1017	16.3 (avg of 3 runs, range 15–18)	21
Kimi K3.1 Pro	13	20
Meta Wonder	7	28
Grok 4.7	12	16
Kimi K3	9	10
GLM 5.3 Flash	5	5

Two details in that table deserve a second look. Deepseek actually beat Daybreak on the Nova scenario, 16.3 to 15.7. And Meta Wonder is a strange case: weak on Nova at 7, but second-best on Lighthouse with 28 of 31 objectives. Capability profiles clearly aren't uniform across attack types.

Takeaway: the open-versus-closed gap in offensive cyber skill is now a matter of degree, not kind — and the open side is far cheaper.

Why did short tests get the answer wrong?

Runtime turned out to be a hidden variable. Deepseek scored zero results through its first 25 million tokens, and it made most of its progress only after crossing 100 million tokens. Most standard evaluations don't give a model anywhere near that much room to work.

The implication is uncomfortable for anyone relying on quick benchmark runs. Short tests underestimate the real potential of open models, sometimes to the point of scoring them as incapable when they're merely slow. If you evaluate a hacker agent the way you evaluate a chatbot, you'll draw the wrong conclusions about risk.

Takeaway: agentic cyber evaluations need long runtimes, or their numbers say more about the test setup than the model.

What do other studies show?

The Vals numbers aren't an outlier. The UK's AI Security Institute (AISI) published an analysis in July that reached a similar conclusion from a different angle. According to AISI, the capability gap between open and closed frontier models in cyber has shrunk to four to seven months. For most of 2025, that gap stood at six to ten months. Open models are closing distance at a measurable rate.

Cost figures from AISI tell the same story as Vals. A cyber-range run with Deepseek V4-Pro costs $1.19, compared with $85 for Anthropic's Opus 4.5. That's a 70-fold difference for tasks in the same domain. AISI also found that the safety measures on the open models proved largely ineffective — a finding that echoes Vals's observation that guardrails weren't what held Deepseek back.

Anthropic's own recent study adds a third data point. It showed that the freely available model GLM-5-3 can build exploits at nearly the same level as Anthropic's internal top model, Mythos. When a lab's internal research concludes that an external, downloadable model nearly matches its best proprietary system on exploit development, the trend is hard to dismiss.

Takeaway: three independent sources — Vals, AISI, and Anthropic — now converge on the same finding from different directions.

Why does this matter beyond the benchmark?

The policy stakes are straightforward. Once a model's weights are public, no provider can control or restrict how it's used, as Vals researchers point out. Closed providers can throttle access, add usage restrictions, or pull a model entirely — as OpenAI does with Daybreak, which sits inside a restricted program. Open-weight models offer no such lever. Anyone can download them and run them for as long and as often as they like, which the runtime findings suggest is exactly what unlocks their potential.

The defense side of the ledger is the one genuinely hopeful note in the report. Vals points out that the same findings create an opportunity for defenders, because the very models that exploit vulnerabilities can help companies find and fix them first. An agent that can chain an intrusion across two simulated companies can, in principle, be pointed at a company's own network to surface the same weaknesses before an attacker does.

Vals expects attacks on real networks to increase in the coming months. Given a model that costs $3 per run, clears 72.8 percent of objectives solo, improves sharply with longer runtimes, and ships without usable safety restrictions, that forecast doesn't require much speculation. The practical question for security teams is no longer whether AI-driven intrusions are feasible — it's whether they find their own vulnerabilities with these tools before someone else does.