Sakana AI's MLR Catches 73% of Core-Claim Errors in Peer Review

Sakana AI's new peer review agent caught 73.43% of planted core-claim errors across 257 research papers, the company reported October 10 in a paper on arXiv. The result runs roughly five times better than the best baseline system tested, and it suggests a sharper role for AI in scientific publishing.

The system, called Multi-Layered Review or MLR, reads papers the way a careful human reviewer would. It also misses things a careful human reviewer would catch, and the gap between those two truths tells the real story.

What does MLR actually do?

MLR runs three agents on off-the-shelf Anthropic models that anyone with an API key can call. A cheap Claude Haiku 3.5 reads the appendix first and pulls out experiments and implementation details. A Claude Sonnet 4 agent then places the paper against prior work using web search. A second Claude Sonnet 4 agent finishes the job with a three-pass prompt chain inspired by Stanford's Keshav Three-Pass Approach, the same method grad students have leaned on for decades.

Pass one sketches a high-level outline. Pass two reads in detail and flags weaknesses, assumptions, and gaps. Pass three merges all agent outputs into a familiar review format with Strengths, Weaknesses, Questions, Recommendation, Score, and a To-Do list, the shape most reviewers know cold. The full three-pass chain produces the kind of structured review a program committee expects, complete with a numeric score, a recommendation, and a list of follow-up questions for the authors. The PDF goes in directly, so figures and equations survive the round trip, which matters for math-heavy fields.

How did Sakana actually test it?

The team built what it calls a Contradiction Benchmark, a way to plant errors in real papers and watch reviewers try to spot them. Researchers collected 257 CC-licensed papers from ACL, AISTATS, CVPR, ICML 2025, and NeurIPS 2024, then injected 1,164 deliberate contradictions across them.

Gemini 2.5 Pro built a knowledge graph of each paper's claims, evidence, and methods. The graph's distance from a main claim set severity. Distance zero meant the contradiction hit a core claim. GPT-4.1 then rewrote one node per distance into a contradiction, yielding the 1,164 data points. An o3 judge scored every review ten times and hit 99.9% accuracy on clean papers, which gave the team confidence in the headline numbers.

How much better is MLR than the alternatives?

On core claims, MLR dominated. With four reviews, it caught 73.43% of distance-zero contradictions, while the best baseline, AgentReview, caught 14.81%. A single MLR review still found 60.79% of planted errors, more than four times the best single-review baseline tested.

The team ran an ablation to separate model from design choices. Swapping GPT-4.1 for Claude Sonnet 4 inside the LLM-Review system lifted distance-zero detection from 14.56% to 35.40%, a 20-point swing from a single model swap. MLR's three-agent design added roughly 25 more points on a single review, evidence that the architecture carries real weight alongside the underlying model. Both choices matter, and the gains stack.

Accuracy fell as node distance grew, which supported the severity scoring Sakana used throughout the paper.

What happens when MLR meets real retracted papers?

The win shrinks fast, and the tone changes. On 211 real retracted papers drawn from the WithdrarXiv-Check dataset, MLR scored 26.07% on "similar" matches and 16.11% on "exact" matches, while the strongest baselines reached 18.48% and 9.00%.

MLR still led. The gap, though, tells a sobering story. Planted errors in benchmark papers follow clean logic that a careful reader can catch. Retracted papers hide their flaws in the wild, mixed into real prose, real figures, and real citations. The benchmark tests reading; the retraction test reads like a stress exam.

Does MLR agree with human reviewers?

On scores, mostly. On ICLR 2025 submissions, MLR's predicted scores reached a Pearson correlation of 0.586 with human scores, the Sakana paper reports, while the human-to-human reference sat at 0.742. On ICML 2025, a rival system called the AI Reviewer edged ahead at 0.439 versus MLR's 0.429, a statistical tie that says little about which is "better."

On focus, no. MLR stresses validity and experiments. Humans weigh clarity and novelty more heavily, a pattern the authors call a complementary perspective rather than a replacement. It is careful word choice for a paper that could otherwise be read as a replacement pitch.

What does it cost to run?

About $0.47 per review, excluding the optional literature agent. MLR uses 189,062 input tokens, roughly half of the AI Reviewer's 403,654. A cheaper single-prompt variant cut cost by about two-thirds, and its detection dropped only 3.5 points on a benchmark subset, an attractive trade for budget-constrained venues.

No GPU is required. No fine-tuning either. The system runs on API models anyone can call, which lowers the bar for replication and raises the bar for what counts as a serious contribution.

What still trips MLR up?

Hidden prompt injection. Every AI reviewer Sakana tested still falls for it, a problem the paper flags but does not solve. An adversary who plants instructions inside a paper can still steer the agent's verdict, and the system has no clear defense.

The retraction result is the other tell. Sixteen percent exact matches on real bad papers means MLR misses 84% of known fakes, a humbling number for anyone who wants to deploy these systems at scale. A useful copilot, not a gatekeeper, and Sakana's own numbers make the case.

Why does this matter now?

Peer review is buckling under the weight of its own success. Submissions to top venues keep climbing. Reviewer pools stay flat. AI tools have flooded in as fast patches that often flatter authors or parrot other reviews, and the field has struggled to set a clean yardstick for what good looks like.

Sakana's framing shifts that question, asking researchers to grade AI on what it catches, not how human it sounds. The same week, arXiv itself began piloting AI-assisted checks for problematic content, a sign that platforms want machine reviewers as a first filter before humans ever read. The pieces point at a future where most papers get scanned, flagged, and only then sent up the chain.

Sakana's code lives behind an email request for now, the paper's comparison table shows, while the benchmark itself is open. That balance suggests a research tool, not a product launch, but the 73.43% number will draw a crowd of follow-up papers and a few uncomfortable editor emails before the next TMLR cycle.