Harvard Study: Benchmark-Higher Claude Fable 5 Lifted Merge Rate Just 1.4 Points
A Harvard study of 297 developers found access to the higher-scoring Claude Fable 5 raised GitHub merge rates by just 1.4 percentage points, a statistically insignificant gain.

Why it matters
- Access to Claude Fable 5, which scored nearly 11 points higher than Opus 4.8 on SWE-Bench Pro, raised the seven-day merge rate by only 1.4 percentage points — not statistically significant.
- Five Claude model upgrades in 2026, covering 1,208 developers and more than 15,000 code submissions, produced no significant gain in merged pull requests or weekly completed changes.
- The study, titled 'Higher-Scoring Coding Models Did Not Get More Work Done,' tracked 297 Fable 5 users against 8,908 Opus 4.8 developers during the 19-day White House ban in June.
- Across the five upgrades, the merge rate actually fell slightly, by an estimated 0.9 percentage points.
- Developers on newer models completed just 2.4% more accepted submissions per week, a difference too small to establish a reliable improvement.
Access to Claude Fable 5, a model that scored nearly 11 points higher than its predecessor on the SWE-Bench Pro coding benchmark, raised the share of developers' code changes merged into GitHub projects by just 1.4 percentage points. A new Harvard University study calls that difference practically a rounding error and not statistically significant.
The paper, titled Higher-Scoring Coding Models Did Not Get More Work Done, tracked 297 developers who used Fable 5 before, during and after a 19-day White House ban on the model in June, and compared their output with that of 8,908 developers who continued using the older Claude Opus 4.8. The finding strikes at the core assumption of the AI coding market: that benchmark improvements translate into more finished work.
Why does a 1.4-point gap matter?
The gap matters because of its size. Fable 5 had posted a benchmark advantage of nearly 11 points over Opus 4.8 on SWE-Bench Pro. In real repositories, that advantage collapsed to a 1.4 percentage point difference in the proportion of pull requests merged within seven days.
The paper states the result plainly: 'Results show that having access to Fable 5 changed the merge rate by only 1.4 percentage points, and the change was not statistically [significant]. We also analyzed five of the eight model upgrades in 2026 and found that the merge rate did not rise after these upgrades [either].'
The researchers continue: 'Similarly, developers did not get more pull requests merged per week. These results show that gains on current coding benchmark did not really translate into more finished work.'
What made the experiment possible?
The White House ban on Claude Fable, imposed for 19 days in June, created an unusually clean natural experiment. Developers submitting pull requests to GitHub had to downgrade to less powerful versions of Claude.
The data was freely available. All versions of Claude co-sign the pull requests they help generate, through a 'Generated with Claude Code' notice automatically included in PR descriptions, and that information is searchable via the GitHub API. Where the specific model name had been copied from the commit history into the PR description, the researchers could establish which version of Claude produced the code.
The study divided the 297 Fable 5 users' pull requests into four periods:
- Before the model's June 9 release
- During its three days of availability
- Throughout the 19-day suspension
- After access was restored on July 1
The 8,908 developers identified as using Opus 4.8, with no recorded use of Fable 5, served as the control group. The comparison assumed that, absent the ban, both groups would have experienced similar changes in merge rates. The statistical analysis accounted for differences between individual developers and day-to-day fluctuations in GitHub activity.
Pull requests submitted around the Fable 5 suspension and subsequent Claude upgrades were collected between November 2025 and October 2026, with at least four weeks allowed for the latest submissions to be merged.
What did the five other upgrades show?
A wider analysis examined five of the eight Claude model upgrades in 2026, covering 1,208 developers and more than 15,000 proposed code changes. The results matched the ban experiment.
No statistically significant improvement appeared in the rate at which submitted code was merged. The rate actually fell slightly, by an estimated 0.9 percentage points.
Developers using newer models completed just 2.4% more accepted submissions per week — a difference too small to establish a reliable improvement. Even where benchmark scores had risen by between 5 and 18 points, the researchers observed no significant productivity gains.
The newer models also performed no better in GitHub's automated code checks. Failure rates remained essentially unchanged.
Beyond the headline merge-rate metric, the researchers ran additional checks: pull requests merged within an hour, PRs closed without being merged, the number of comments received, changes subsequently reversed, and failures in automated checks.
Why do benchmark gains vanish in real work?
The paper offers a hypothesis for the disconnect between benchmark scores and shipped code: 'We hypothesize that the main reason Fable 5's gains on benchmarks did not translate to real-work productivity is that the tasks developers give agents are already within what the older model can do.'
The authors also flag the structural difficulty of measuring model effects in the wild. 'People who use new models first may be more experienced, so when we compare productivity across developers, we cannot tell if the difference comes from the model or from the people using it. Even if we observe the same people, we do not know if they give the same tasks to different models.'
Task allocation can distort results in the other direction. 'For example, some people might use Fable 5 for the hardest tasks and switch to lower-tier models for easier tasks. If they give the harder tasks to the strongest model, the strongest model may look less productive on almost any measure we choose, since harder tasks take longer and are less likely to be merged.'
The paper further contends that earlier studies were skewed because most code changes came from a small number of projects, each with its own rules for accepting contributions. Those differences made it hard to attribute productivity gains to model upgrades.
The stakes extend beyond engineering teams. The authors warn: 'If regulators and buyers judge how much stronger a model is by its benchmark [score], the difference they rely on will likely not show up in user productivity.'
How reliable is GitHub merging as a metric?
The paper acknowledges that GitHub merges are an imperfect measure of productivity. The ban nonetheless offered a rare chance to assess model switching under real working conditions — in a sector that attracts huge global interest while operating, as the researchers note, under maximum corporate opacity.
The measurement problem runs deeper than any single metric. Benchmark test data tends to leak into the training data of the next model generation, rendering the benchmark useless for its intended purpose. The speed of that contamination, the study's authors argue, breaks the template for corporate benchmarking and for consistent scientific testing that depends on previously established metrics.
The problem predates large language models. The paper points to a decade of computer vision research papers still adhering to long-outmoded standards such as ImageNet, simply because of the age of their provenance.
The study does not argue that model improvements are illusory. It argues that current benchmarks do not measure the work developers actually assign, because those tasks already fall within the capabilities of older models. Simon Wilson's informal 'Pelican riding a bicycle' challenge for vision language models illustrates the same principle from the opposite direction: models perform poorly when asked to create content absent from their training data — though, as one recent Hacker News commenter observed, that test is now itself compromised, because the pelican-on-bicycle data has entered training corpora.
What does this mean for buyers and regulators?
The findings present developers in a favorable light: they adapt to whatever system is running and perform at roughly the same speed with the latest model or the last one, optimizing to durations and deadlines regardless of the tools at hand.
For buyers, the message is direct. A benchmark score, on its own, is not evidence that a model will produce more work. The paper's conclusion stands as the operational takeaway: 'Our findings add evidence from real work to what prior work has warned about: a benchmark score means little to people choosing a model without evidence that it measures the work it is used to [describe].'
The study, first published in November 2026, also poses two additional research questions beyond its central finding, addressed in the source PDF.
Source: unite.ai
More from Rebecca Stone
Show full bio
Correspondent covering companies, deals and strategy in artificial intelligence and machine learning at AI In Context.
728 articles
Related articles
- Anthropic Launches Fable 5.2 and Mythos 5.2: Faster, Cheaper, Sharper on Security
- AI coding agents fully complete only 26.4 percent of real-world tasks
- Anthropic's Fable 5.2 completes 41 percent of real freelance jobs
- Anthropic's Claude Opus 5.5 costs 40% less, hits Fable 5.1 performance
- Microsoft's MAI Code 2 hits 96% of Claude Opus 5.5 at 5% of the cost