Coding-agent harnesses triple costs without lifting pass rates

By Marcus Bennett

Coding-agent harness choice could make one task cost up to three times as much, while most pass-rate differences disappeared within rerun noise in a Shandong University study by Liu and Han.

The finding matters because makers tout score gains from harnesses such as Claude Code and OpenCode. Liu and Han noted that new harnesses ship every day and leaderboards mix them freely, although one unchanged setup can vary by as much as six points on SWE-bench Verified.

Earlier studies cited by Liu and Han reported gaps above 20 points, usually from single runs. The authors said benchmark wiring also differed across harnesses, making those comparisons less reliable.

A coding agent pairs a language model with a harness: the system prompt, tools and context management that let the model work inside a code repository. The paper’s title states the result: “What does a harness buy? Mostly tokens.”

How did Liu and Han test the harnesses?

Liu and Han ran five models through three harnesses in their shipped versions and added repeated runs to estimate noise.

• Harnesses: The authors used Claude Code, OpenCode and mini-SWE-agent. mini-SWE-agent exposed only Bash and omitted context management. Claude Code and OpenCode added specialized file and search tools, long system prompts and automatic history compression.
• Models: Their five primary models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash and HY4-Preview. Claude Opus 5 served as a reference model only in Claude Code.
• Workload: Their tests covered the 45 hardest tasks in SWE-bench Verified and a separate pool of 447 additional tasks. Several configurations ran twice so Liu and Han could measure run-to-run variation.
• Access: Every scored run took place offline. Liu and Han ran some earlier trials with web access before identifying a strong pattern of benchmark leakage.

In the discarded web-enabled trials, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task. Claude models almost never did, according to Liu and Han.

The researchers said they make no claim that any maker trained on the test data. They nevertheless disabled web access for every remaining run because the pattern was strong enough to threaten benchmark validity.

Do different harnesses solve more tasks?

On the 447-task pool, Claude Code and mini-SWE-agent finished less than two points apart for both models tested. The researchers treated them as statistically equivalent within a five-point band.

The 45 hardest tasks produced no clear winner for Liu and Han. The researchers expected the largest harness effect there, but harnesses differed by two to five solved tasks per model, while a rerun of the same setup changed the count by as much as three. mini-SWE-agent led sometimes; Claude Code led others.

Median outcomes changed on 13 percent of tasks between reruns and on 13 percent between harnesses. Swapping models raised that figure to 22 percent. The authors said the changing set of winning tasks resembled a coin flip and called the harness gap a fresh draw of the same noise.

With only 45 tasks, Liu and Han calculated that even a 13-point gap appeared only half the time. With 447 tasks, gaps of about five points became reliably detectable. Zhipu AI advertised a 2.4-point lead over Claude Code, a margin below both thresholds, as were many other claimed improvements.

Why does OpenCode fall behind?

OpenCode recorded the only measurable harness effect, but it was a steady deficit of as much as nine points on the 447-task pool. Liu and Han found that about half of that gap traced to an output limit.

When a model exceeded the limit during mid-step reasoning, Claude Code prompted it to continue. OpenCode ended the step, and the attempt usually failed, according to the researchers.

Adding a Claude Code-style continuation prompt to OpenCode did not help. The same task rarely produced the same overflow on a retry, Liu and Han found.

For the unexplained remainder, the authors offered only an observation: models stopped working sooner under OpenCode. On locally run models, OpenCode also imposed a much tighter time limit than Claude Code, an uncontrolled variable in the experiment.

mini-SWE-agent exhausted the context window because it lacked context management, mainly on tasks where the other harnesses also failed. OpenCode’s search tools tried to download a binary file on first use and broke in the offline setup.

Why can one harness cost three times as much?

With the model, tasks and price list fixed, one task could cost up to three times as much depending on the harness. Tool outputs did not explain the spread; all three harnesses returned similar amounts of text per call, Liu and Han found.

The cost difference came from system prompts and tool descriptions. Both were resent at every step, running about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode and under 1,000 for mini-SWE-agent.

Claude Code also retained the model’s reasoning output in the conversation history, a behavior the researchers identified through reverse engineering.

After the first call, context grew at similar rates under all three harnesses. The system-prompt size set the baseline cost, and each additional step multiplied it, according to Liu and Han.

The dominant factor varied by model. With GLM-5.3-Flash, Claude Code cost nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needed enough extra steps to reach the same total cost.

Lower prices for cached tokens reduced the gaps without changing the ranking, according to the study.

Does thinking change the result?

Thinking was the only case in which harness choice changed the outcome, Liu and Han found. Without thinking, mini-SWE-agent with Qwen3.6 dropped 11 points, while Claude Code and OpenCode stayed within noise.

HY4-Preview also lost 11 points in Claude Code without thinking. The authors said dependence on reasoning appeared tied to the specific model-harness combination, but their test grid was too small to explain why.

Deepseek’s own harness did not score better with its own model than the other harnesses did, yet it cost the most, according to Liu and Han.

After the researchers removed tools from Claude Code, Claude Opus 5 delivered identical results. A year-old version solved a similar number of tasks at less than half the cost because Extended Thinking was not on by default then.

What can the study establish?

The study relies mainly on SWE-bench Verified, with a small cross-check on Terminal-Bench 2.1 showing the same pattern. Its closed-source top model, Opus 5, ran only in Claude Code.

Liu and Han’s conclusion was blunt: “fewer ways to lose a task, and a bill, not pass rate.”

The researchers recommend publishing a rerun of the same setup alongside every harness comparison. They call that the only way to separate a measured difference from noise, and they published the data and analysis scripts on GitHub.

What should harness comparisons report?

The Decoder also cited three related results: a simple Markdown file beat a complex skill system for AI coding agents; an earlier study found that framework choice could nearly triple costs for the same model; and Nvidia recently cut nearly half its coding-agent token costs through better agent steering.

The next credible harness comparison will need repeated runs of the same setup. Until vendors publish them, Liu and Han’s evidence favors evaluating cost and failure handling alongside headline pass rate.