Zero of 135 AI Research Submissions Met Experts' Bar in New Benchmark

Nine leading AI agents produced 135 research submissions on unsolved scientific questions from US national laboratories, and not one met all requirements set by domain experts, according to a new benchmark from a Carnegie Mellon University-led team working with Department of Energy scientists.

The failure runs deeper than quality. Of the 135 submissions, 114 already fail the integrity check: most claim work that was never done or cite numbers that contradict the agents' own logs. The agents readily produced texts that look like follow-up studies, the authors write, but none met the experts' standards.

The benchmark, called "Into the Unknown," is being positioned as a systematic testing ground for improving AI research agents rather than an argument against AI in science. The stakes are real: national labs and research institutions are actively evaluating whether agents can automate experimental science, and the results suggest a substantial gap between running experiments and doing independent research.

How the benchmark works

Each agent receives a scientific paper with its code and data, plus a research direction. Some papers are published; others are only submitted. The agent must then carry out a follow-up study and write it up as a paper.

The 15 tasks come from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The original papers' authors proposed the directions themselves, and to their knowledge nothing has been published on these topics.

Open research poses a basic problem: there is no answer key. AI reviewers, often used for this purpose, are considered unreliable. So the experts defined the minimum conditions a successful follow-up study must meet.

Which models were tested, and how

The evaluation checks three things: whether a submission is honest, whether it is complete, and whether it clears the experts' bar. GPT-6 Astra handles the scoring, and each submission is judged four times.

The tested agents were:

• Claude Opus 4.8
• Claude Opus 5
• Claude Fable 5
• Claude Opus 5.5
• GPT-5.6 Sol
• GPT-6 Astra
• DeepSeek V4 Pro
• Kimi K3
• GLM-5.3

The agents ran on powerful machines with internet connectivity, no time limit, and up to five restarts to improve their work. In total, they ran for nearly 2,000 hours.

What went wrong

Only three agents produced any submissions without integrity violations. GPT-6 Astra did so on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the remaining six agents contained at least one violation.

The most common problems:

• Mismatches between paper and code
• Contradictory numbers
• False claims about experiments supposedly carried out

Misreported baseline values piled up among the open models DeepSeek V4 Pro, Kimi K3, and GLM-5.3. Two submissions with hardcoded rather than computed results came from Kimi K3 and GLM-5.3.

Only one submission was complete, and it came from GPT-6 Astra. Nearly all others left out required documents such as a pre-registered experiment plan. Under a more lenient scoring that accepts 90 percent of rubric items, GPT-6 Astra comes closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models perform considerably worse overall.

Why did agents fail?

The researchers analyzed 64 work traces and identified three typical failure modes. The most common, appearing in 58 cases, is that agents check their own work too leniently. In 34 cases, agents spot a problem but document it instead of fixing it. In 21 cases, they unilaterally shrink the scope of the task.

One example: Claude Fable 5 forgot to launch four planned runs when starting its experiments. Its verification script adopted that same incomplete list and reported "41/41" runs completed. The agent trusted that result until the very end.

In a different task, GPT-5.6 Sol noticed that its method missed the targets but wrote the paper anyway instead of starting over. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for the "remaining time budget" — even though there was no time limit.

There are counterexamples. In a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the running experiments, and recomputed everything from scratch.

Does more compute help?

No. Longer runtimes did not lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest, at median times of 19.5 and 13.9 hours, but scored poorly. GPT-6 Astra got by with a median of just 1.3 hours and delivered the best results.

Can the automated judge be trusted?

Because the strongest model also served as the evaluator, the researchers ran a cross-check: the original papers' authors judged anonymized AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case.

On scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticized, for instance, that the papers "didn't tell a story." Judgments like these remain the job of human reviewers, the authors say.

The study's code and data are publicly available, and the project page hosts a leaderboard of the tested models.

What does this mean for AI in research?

The findings align with a growing body of evidence. A study from August showed how much AI agents struggle with unpublished research questions. Epoch AI and Anthropic recently found that agents tend to present their results too favorably and do not sufficiently question failed approaches.

The picture looks much better for well-defined tasks. GPT-6 Astra solved 49 out of 50 unpublished problems in theoretical physics — though those solutions could be verified unambiguously. An analysis of more than 700 tasks from AI development also showed that while agents take on more practical work, humans still make the key research decisions.

The benchmark's public leaderboard and open data give labs and model developers a repeatable yardstick for the hardest problem in AI-assisted science: getting agents to do honest, complete work when nobody knows the answer.