AI coding agents fully complete only 26.4 percent of real-world tasks

The best AI coding agent tested by researchers from Alibaba and several Asian universities fully completes only 26.4 percent of real-world tasks, despite older benchmarks showing scores above 99 percent. The finding comes from Co-Design Bench, a new benchmark built from actual user change requests, and it reframes what near-perfect leaderboard results mean for developers who deploy these tools in production.

GPT-6 Astra, the strongest of the ten models tested, meets about 70 percent of a task's requirements on average. Yet the output counts as fully correct in only 26.4 percent of cases. In most tasks the agent does the bulk of the work but leaves at least one detail unfinished, which means anyone using these tools almost always has to clean up by hand.

Why did researchers build a new benchmark?

The authors justify Co-Design Bench by pointing to the saturation of existing tests. When models can run, check, and fix their own code, they exceed 99 percent on older benchmarks, and differences between models can barely be measured. A test that every contestant aces no longer ranks anyone.

Co-Design Bench draws on real requests from actual coding sessions instead. The researchers deliberately favored cases where users had to rephrase their request multiple times or roll back changes. The result: the benchmark contains mainly tasks that already gave AI agents trouble in practice. Tasks range from small tweaks — pulling an amount from a receipt and updating every dependent total — to full-scale project overhauls.

The agents must rework existing web apps based on user requests. Input is not limited to text; requests can arrive as images, audio recordings, videos, or documents, mirroring how real users actually communicate what they want changed.

How strict is the grading?

Grading matches the real-world framing. One AI judge tests the finished app in a browser. A second judge then re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app. An agent cannot pass by faking functionality.

That double-check mechanism matters because flawed test environments can let agents pass tasks without solving them, a failure mode the authors explicitly guard against.

Which models won, and which lost?

Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max take the next spots. Grok 4.6 lands in last place among the ten models tested.

One anomaly stands out. On larger overhauls, the much smaller Qwen3.8-27B climbs to second place. The authors do not derive a general advantage of the smaller model from that single result.

Performance drops across all models when a task combines more types of changes — layout, design, interaction, and content at once. This matters because intertwined requirements are the norm in real projects, not the exception. Agents that handle single-dimension changes well falter when a user asks for a redesign that touches everything.

Extra compute does little to fix the weakness. GLM-5.2 burns through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse. Spending more does not buy correctness.

How much does the harness matter?

In a separate experiment, the researchers ran the same models in different harnesses — the programs through which an agent reads, writes, and executes code. The spread was large. Qwen3.8-27B scored 75 percent or just 56 percent on larger overhauls depending on the environment. Which of two models came out ahead also depended on the environment.

The implication reaches every developer choosing a coding agent. Leaderboards that list only the model tell part of the story. The authors recommend always reporting model and working environment together. For companies picking a coding agent, that means testing both components as a pair before committing.

The evidence on harnesses is not uniform, however. A recent study on coding agents found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.

What are the benchmark's limits?

The new test has acknowledged constraints. It evaluates one fixed request per task; follow-up questions or mid-task corrections from users are not part of the test. Real coding sessions rarely stay that static. The AI judges can also make mistakes, though the two-judge design reduces the risk.

The gap between benchmark scores and real-world usability has independent support. A study by the research organization METR found that the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified. Co-Design Bench's 26.4 percent full-completion figure for the best model sits in the same sobering territory.

What does this mean for buyers and developers?

The stakes are commercial as much as technical. Companies evaluating coding agents on saturated leaderboards may overestimate what the tools deliver, then absorb the cleanup cost in engineering time. A tool that finishes 70 percent of requirements still leaves a human to find, fix, and verify the remaining 30 percent on nearly three out of four tasks.

The paper's practical guidance is concrete: test the model and its harness together, on tasks that resemble the messy, multi-type change requests your teams actually receive. Until benchmarks like Co-Design Bench become standard, the 99-percent scores on older tests describe a world most developers will not recognize — one where the request never changes mid-task and every feature either works or visibly fails.