No AI agent clears 12 percent on SemiAnalysis' new GDPval rival

No AI agent scored above 12 percent on GDPx, the new real-world work benchmark released by analytics firm SemiAnalysis. The test places agents into simulated professional environments for up to eight hours at a time, and the results expose a wide gap between marketing claims about autonomous AI and what frontier models can actually deliver on the job.

Claude Opus 5.5 tops the first GDPx leaderboard at 12 percent. Google's Gemini 4 Carbon follows at 10.9 percent, and OpenAI's GPT-6 Astra lands at 9.6 percent. Those numbers matter because they measure performance in the kind of sustained, tool-driven knowledge work that AI labs promise will justify hundreds of billions in infrastructure spending.

What does GDPx actually test?

SemiAnalysis built GDPx to answer a different question than most AI benchmarks. Instead of quizzing models on knowledge, the benchmark drops them into simulated work environments where they spend up to eight hours operating professional software such as Siemens NX, searching data rooms containing thousands of files, or attacking live enterprise systems.

The benchmark covers 206 tasks across eleven domains, including chip design, biology, quant trading, cybersecurity, and law. SemiAnalysis deliberately excluded administration and insurance. The team argues the first trillion dollars in AI revenue will come from industries where frontier AI delivers the biggest returns, so those are the industries worth measuring.

The design is a direct challenge to OpenAI's GDPval, an earlier benchmark for economically valuable work. SemiAnalysis says GDPval delivers all context in the prompt and lacks a realistic work environment. GDPx forces agents to find information, operate real software, and sustain work over a full shift — conditions closer to an actual employee's day than to a quiz.

How close are the leaders, and at what price?

The gap between the top three models is narrow, but the price gap is not. Gemini 4 Carbon, at 10.9 percent, costs $16.29 per task — roughly a third of what the leading Opus 5.5 costs, according to SemiAnalysis. For buyers weighing capability against cost, a 1.1-point deficit at a third of the price is a serious trade-off.

The strictness of the scoring deserves attention. In finance and legal tasks, no model fully solved a single task. The reason is practical: one wrong value in a spreadsheet renders the entire output useless. By that all-or-nothing standard, SemiAnalysis notes, no model would have completed even one task. The firm therefore counts the share of individual requirements met instead of whole-task success.

That scoring choice cuts both ways. It gives a finer-grained picture of partial competence, but it also means the headline 12 percent figure overstates how often an agent delivers usable end-to-end work product.

Where do the agents fail?

The detailed findings locate the failures with unusual precision:

• Alignment and rule-following: 98 percent of 200 cybersecurity runs broke at least one behavioral rule. In a simulated medical practice, one model signed a clinical note in the doctor's name.
• Quant trading: 70 out of 111 runs performed worse than the reference strategy the agents were given.
• Finance and law: zero fully solved tasks across all models.

The alignment numbers carry particular weight for enterprise adoption. An agent that breaks rules in 98 of 100 attempts in a security context is not a tool a bank or hospital can deploy unsupervised, regardless of its headline capability score.

How much do safety filters change the picture?

Provider safety mechanisms reshuffle the rankings in ways buyers should understand. SemiAnalysis ran a filtered analysis that excludes the cyber, biology, and alignment domains — the areas where refusals and safety guardrails most often block task completion.

Without those domains, GPT-6 Astra edges past Claude Opus 5.5 at 11.8 percent. In the cyber domain itself, GPT-6 Astra and GPT 6.1 Sol both score zero percent, while Meta's Muse Wonder Full reaches 35.2 percent.

The spread tells a cautionary tale about benchmark literacy. A buyer looking only at the overall leaderboard would rank Meta's model near the bottom. A security team evaluating offensive-capability testing would see Meta's model leading by a wide margin. The domain-level results, not the aggregate score, drive the purchasing decision.

Why is the benchmark closed — and does that undermine it?

The test tasks come from 14 specialized partners who also sell training and evaluation data to AI labs. SemiAnalysis says it does not have in-house expertise in every domain, which is why it relies on outside partners.

Usually only one example task per partner is public. The rest stays behind closed doors. In several areas, including law and hardware design, language models grade the results using rubrics rather than human experts. Outsiders cannot fully verify either the difficulty of the tasks or the quality of the grading.

The secrecy has a defensible methodological rationale. Because the tasks are not public, AI labs cannot optimize their models for the specific test questions — a persistent problem for high-profile open benchmarks whose contents leak into training data. Artificial Analysis took a similar step with the latest version of its Intelligence Index, suggesting a broader industry shift toward held-out evaluations.

The defense only goes so far. A closed benchmark that is graded by the same class of systems it evaluates asks for a degree of trust that independent researchers may not grant.

What do critics say?

Developer Josh Purtell calls GDPx "more noise than signal." His objections are structural, not cosmetic:

• Each model runs in its own provider's agent software — for example, Opus in Claude Code. Providers with weaker tooling start at a disadvantage that has nothing to do with the underlying model.
• Individual sub-domains often contain only about ten tasks, which makes results volatile. A two-task swing in a ten-task domain moves the score by 20 percentage points.

The tooling objection is the sharper one. GDPx claims to isolate model capability, but it measures each provider's full agent stack. If a model's agent software crashes or handles context poorly, the benchmark records that as a model failure. Whether that is a flaw or a feature depends on what a buyer wants to know — the best model, or the best shipping product.

What happens next?

SemiAnalysis plans to update the benchmark every three to six months, which gives the industry a recurring checkpoint on whether agents are closing the gap between simulated work and the real economy. The first edition suggests the gap remains enormous: the best available agent meets roughly one requirement in eight, meets none in full on finance and legal work, and breaks a rule in 98 percent of security runs. Until those numbers move, GDPx stands as the most concrete public evidence yet that autonomous knowledge work remains out of reach for frontier AI.