AI Reviewers Fixate on Paper Appendices, Study Finds
A 2,000-paper study found that newer OpenAI models drew 50%–70% of criticisms from appendices, while human reviewers focused on the main text. The bias persisted across four access methods.

Why it matters
- The study examined 2,000 papers, including 1,200 ICLR papers from 2021–2026 and 800 recent conference papers.
- Newer GPT models drew 50%–70% of criticisms from appendices, while GPT-5 drew less than 30%.
- All 200 LLM-generated ICLR 2027 reviews included appendix criticism; only 159 human reviews sampled from 2021–2024 did.
- Across four access methods, appendix material appeared in 62.2%–73.7% of criticisms.
- Anderson's Angle published its analysis on November 16, 2026.
A new U.S. study of 2,000 scientific papers found that newer OpenAI models drew 50% to 70% of their criticisms from appendices, compared with less than 30% for GPT-5. Anderson's Angle summarized the work, “AI Reviews Are Biased Towards the Appendix,” which reported that human reviewers assigned fewer than 10% of weaknesses to appendix material.
The stakes are practical: automated peer review must test a paper’s central claim, not reward exhaustive criticism of supporting pages. The paper identifies GPT-6.1-Sol and GPT-6-Astra, both released in 2026, as the clearest examples of the shift. Anderson's Angle first published the analysis on Monday, November 16, 2026.
“The difference between the generations is large. For GPT-5, only a small share of weaknesses relate to the appendix. Both newer models, GPT-6.1-Sol and GPT-6-Astra, rely on it much more heavily, and this holds for every conference.
This suggests that the bias towards the appendix is mainly a property of newer models. Even though the new models are an improvement, their bias toward the appendix has grown. Being able to read a paper in detail has not taught them which parts matter most.”
The authors still call both newer models an improvement, but say detailed reading did not teach them which sections matter most. Broader coverage and weaker prioritization form the study’s central warning.
What did the study find?
The pattern appeared in papers from ACL 2026, EMNLP 2026, COLM 2026 and ICLR 2027. Researchers compared AI criticisms with human reviews and GPT-5 output, classifying weaknesses by whether they concerned the main paper or its appendix, the supplementary material placed after the references.
“[Human] reviewers treat the main body and the supplementary material very differently from LLMs. LLMs attribute more than 50% of their weaknesses to appendix-related material, while humans attribute less than 10%.”
At the individual-review level, the paper reported:
“The difference also shows at the individual-review level. Of the 200 papers sampled each year from 2021 to 2024, only 159 in total received a human review with any appendix-related weakness. In contrast, all 200 LLM-generated reviews for ICLR 2027 included at least one appendix criticism.”
AI reviewers were more than seven times as likely as humans to identify an appendix weakness. They were also more than three times as likely to raise criticism confined to supplementary material, even without a stated connection to the main contribution.
The study does not show that every criticism was wrong, only that attention had shifted. AI found more appendix problems, not necessarily more important ones. In plain terms, the models searched harder rather than judging better.
How did the researchers test 2,000 papers?
Researchers tested GPT-6-Astra, GPT-6.1-Sol, Claude-Opus-5.5 and Claude-Sonnet-5.5 from OpenAI and Anthropic. They used Codex and Claude Code, then compared them with API (application programming interface) and web access. The paper calls itself the first study to identify this syndrome.
The corpus contained 1,200 ICLR submissions from 2021 through 2026 and 800 papers from ACL 2026, EMNLP 2026, COLM 2026 and ICLR 2027, with 200 from each recent conference. All 800 recent papers had appendices of at least 15 pages.
The conferences used similar review templates, strict main-paper limits and generally unrestricted appendices after references. ICLR 2027 submissions were already anonymous; researchers removed names, affiliations, emails and identifying links from the other papers. Internet access stayed disabled, so reviews used only the supplied PDF (Portable Document Format).
Each model first produced a conference-style review covering strengths, weaknesses and an overall score, using deliberately simple instructions. Researchers then returned the paper and completed review to the same model and asked it to locate the section behind each criticism. That two-stage design prevented section-mapping instructions from influencing the original review.
About half or more of the criticisms made by all four models came from appendices, and the pattern held across every conference. Criticisms confined to appendices without a connection to the main paper accounted for 17.33% of weaknesses identified by GPT models, versus 10.16% for Claude.
A separate test checked whether access methods changed the result. Using GPT-6-Astra, researchers reviewed 100 ICLR 2027 submissions through Codex, the ChatGPT web interface, an API given the original PDF and an API given extracted text.
Appendix material appeared in 62.2% to 73.7% of criticisms across those four routes. The highest proportion occurred when extracted text arrived through the API, showing that the bias persisted regardless of how the model received the paper.
Codex extracted and examined text through commands, the API received either a complete PDF or extracted text, and the web interface used an opaque document-processing system. The access format changed the details of document inspection, but not the overall pattern. In plain terms, changing how the paper arrives does not reliably change where attention lands.
Why do the models focus on appendices?
The authors blame an equal-weight treatment of text. Large language models (LLMs) do not reliably recognize that the appendix is not normally where a paper’s payload sits. Less-curated, rougher appendices contain more errors to correct, while the main body can read as factual reporting accepted at face value.
“LLMs almost never criticize the opening pages, where a paper frames its problem and argues why it matters. Humans often push back on exactly this framing, while LLMs seem to accept it and move on to the numbers.”
Access format may alter where a model looks, but not the broad pattern. Anderson's Angle reports that Codex does not read the middle of an average-length PDF at all, sampling it later only in “bites” if necessary. Every tested access method still favored appendices.
“Human criticisms fade quickly as the appendix goes on, and almost disappear in its last pages. LLM weaknesses do not fade at all. In fact, LLM weaknesses peak well into the appendix rather than at its start, and they continue even beyond page 30, where human reviewers raise almost no concerns.”
The paper compressed that difference into one line: “For an LLM, a detail on page 25 is as fair a target as the main results table.” Human criticism, by contrast, had almost vanished by page 30.
“Humans expect the main body to stand on its own, so they object when essential sections are pushed into the appendix. LLMs almost never raise this concern.”
The authors interpret that gap as a failure to recognize scientific hierarchy. A claim, experiment or limitation in the main body can determine whether the work supports its conclusion. An appendix correction may not.
Anderson's Angle goes further, describing appendices as an untidy dumping ground for secondary evidence and caveats. In that reading, AI has not become a better critic of significance. It has become an efficient detector of rough material wherever that material sits.
The mechanism may be statistical rather than scientific. In plain terms, the models reward finding something to criticize wherever it appears.
What does this mean for AI-assisted science?
The study warns that AI-assisted research could stall if reviewers spend their attention on minor details while overlooking the main findings. It also records the other side: appendix-focused review can catch real errors, including ones buried on later pages. The unresolved issue is whether those catches compensate for reviews that never engage with the paper’s core.
“LLM reviewers raise the standard, since they leave no part of the paper unexamined. However, effective reviewing also requires judgment about what matters most.
When most of a review’s weaknesses come from appendix content unrelated to the main contribution, the feedback stays on the surface. It does not engage with the core of the paper and does little to help authors improve it.”
The authors acknowledge that no part of a paper should escape scrutiny. Their warning is that scrutiny without priority produces length without usefulness, a problem that grows if model-generated reviews move from assistance toward replacement.
The project began after researchers noticed odd patterns in LLM-assisted or generated reviews of their own submissions. They released the study’s code on GitHub, although the article provides no repository link and describes no tested remedy.
Anderson's Angle also suggests that appendix fixation may influence human reviewers. That claim remains speculative: the human sample contains less appendix criticism, and the study describes no experiment showing that model behavior causes human behavior. Long-term effects remain unknown because the tendency appeared only in the newest models.
The evidence establishes disproportionate attention, not a proven causal chain. In plain terms, the study identifies a bias without yet prescribing a reliable fix.
Are authors and conferences part of the problem?
The commentary argues that AI should not receive all the blame. Authors often place important qualifications, contradictory results and other “gotchas” in appendices, while conference page limits force some evidence out of the main paper. In the worst cases, the main body becomes an index for material deferred to the rear.
Submission rules also limit how much an AI-guided reviewer can reasonably inspect. A reviewer who budgeted ten pages for the main paper can face 30 pages of appendix under the same allocation. Anderson's Angle says conference guidelines are catching up and banning this deferral.
That leaves a structural tension. Conferences need shorter, disciplined presentations as submission volume rises, while researchers want enough space to preserve the context behind difficult results. Neither side can resolve the reviewer’s priority problem simply by moving more material.
The next useful test is whether reviews that rank criticism by its effect on the central claim can retain useful appendix checks without burying authors in secondary detail. Until then, appendix-heavy output should be treated as a warning sign, not proof of deeper scrutiny.
Longer review is not automatically better review. In plain terms, a model can scan 30 pages and still miss the claim that makes the paper matter.
Source: unite.ai
More from James Calloway
Show full bio
News editor covering people, launches and day-to-day practice in artificial intelligence and machine learning at AI In Context.
722 articles