Stanford study finds 44 percent of cited LLM research claims have expired

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found that 44 percent have already expired. Among the 22 findings with an estimated expiration date, the average lifespan of a research result was just 15.7 months.

The study matters beyond academia. Governments, companies and courts increasingly cite published LLM research when they write policy, buy systems or decide what these models may be trusted to do. If nearly half of that evidence base decays within roughly a year and a half, every claim about model behavior carries an implicit expiration date that few papers state.

How the study was run

Queen and Zou examined the claims using models released between July 2024 and September 2026 by eight different developers. They re-ran the original experiments on the current generation of systems and sorted each finding into categories: expired, weakened, holding, or strengthened.

The results break down as follows:

• 44 percent of the 50 findings have expired — the original effect has largely disappeared.
• 34 percent have weakened significantly.
• 16 percent still hold.
• 6 percent have actually gotten stronger.

Decay accelerates with time. After one year, 16 percent of findings had expired. After two years, that number climbed to 39 percent.

What expires fastest?

Claims about what models cannot do expire fastest, the study finds. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. The expired weaknesses include well-known limitations:

• Counting letters in the word "strawberry."
• Arithmetic in unfamiliar number systems.
• Losing information in the middle of long contexts.
• Abandoning correct answers when users push back.

The researchers suspect these errors have a "correct" outcome, which gives developers a clear target to train toward. A model can be fixed on strawberry; there is a right answer.

Claims about preferences and tendencies prove longer-lived. Of the 11 findings that have held or gotten stronger, three involve LLM behavior as evaluators: models prefer their own answers, they rate accepted and rejected scientific papers similarly, and they are swayed by linguistic spin in medical abstracts. Three more concern sycophancy and agreement toward users. The remaining persistent findings include a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.

What happened to hallucinated references?

As a case study, Queen and Zou looked at the fabrication of scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. The Stanford researchers sent the same prompts to current models.

The spread in the results is the story:

• GPT-6.1 Sol fabricated 0 percent of references.
• Claude Sonnet 5.5 fabricated 1 percent.
• Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent of its references in the same test.

The study warns against reading these numbers as an all-clear. On so-called two-hop questions, where a model has to combine two known facts, Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions. The original paper reported 43 percent for ChatGPT. The "Reversal Curse" and contradictory answers to paired time-related questions have weakened too, but they remain detectable.

Do social biases persist?

Social biases present a mixed picture, according to the researchers. The left-leaning political bias persists. Gender stereotypes and implicit association biases have weakened. Biases against speakers of African American English have actually flipped direction in the trend data.

How wide are the gaps between models?

Results vary considerably across models, even within the same release window. Among models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider, nor model size reliably predicts the pattern.

That variance undercuts a habit baked into AI journalism, procurement and regulation: treating "LLMs" as a single entity with shared properties. A claim that expired on one 2026 model still holds on another released in the same quarter.

What do the researchers recommend?

Queen and Zou draw a broad conclusion. Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document the model, version, and date, and publish prompts and analysis code so findings can be re-tested on future models.

The researchers point to a post by Andrej Karpathy that describes how perception gaps about LLMs' actual behavior affect trust in AI and how often people use it. Karpathy, an OpenAI co-founder who has since joined Anthropic, also recently said that currently available models are just a preview of future capabilities.

The study's practical implication runs in the opposite direction of the model releases it measures. As capabilities move, the evidence base moves with them. Findings that inform safety reviews, bias audits and purchasing decisions today may describe systems that no longer exist in 15.7 months — and the researchers argue the field should start printing that date on its results.