METR: Anthropic packed two years of benchmark gains into 12 months

About 80% of Anthropic’s current acceleration in model benchmarks came from AI doing research work, according to METR’s best estimate in a review published Thursday, Nov. 5, 2026.

The nonprofit spent 11 weeks reviewing Anthropic’s internal data. METR’s conclusion was that Anthropic had packed roughly two years of benchmark gains into the last 12 months.

METR found that since October 2025, benchmark scores for Anthropic models had grown at about 1.9 times their earlier rate, with a 95% confidence interval of 1.5 to 2.8 times.

METR staff said they were confident the jump did not come from faster growth in training compute.

The report attached “substantial uncertainty” to the 80% figure, and its supporting methods produced estimates from 1.1 to 2.9 times.

That combination addresses METR’s central causal question: whether faster training compute or AI-assisted research drove the improvement. METR’s staff rejected faster compute growth as the likely driver, then assigned most of the residual acceleration to AI-assisted work.

The review’s reliance on three internal-only models also limits independent verification. Outside researchers cannot submit those systems to the same public evaluations.

McCandlish said Anthropic’s forthcoming analysis would state where the company stands against its Responsible Scaling Policy. That makes the estimate relevant to the lab’s own safety process.

What did METR measure?

Anthropic gave METR benchmark results for unreleased models, training metrics, headcount data and about 15 staff interviews.

The report identifies three internal-only Anthropic models that finished training in 2026. They carry heavy weight in METR’s estimate.

• Model 2 finished training in July 2026.
• Model 3 finished in August or September 2026.
• Model 4 finished in September or October 2026.

The package mixed quantitative model records with staff accounts, giving METR both benchmark outcomes and information about research inputs.

Model 4 scored above 50% on every benchmark it was tested on. That result left METR “somewhat nervous” about whether it could keep measuring progress as models improve.

How strong is the evidence?

Evidence for the broader claim remains soft, and METR’s authors said so.

In October Slack polls, Anthropic staff rated their speedup from the newest internal model at about 12 times.

METR called the polls “only very loose evidence.”

The poll captured staff impressions, while the 1.9-times estimate came from benchmark score trends. METR did not treat the 12-times response as equivalent evidence for the headline rate.

The report says AI still works mostly as an assistant to human researchers. It cannot yet make “highly significant and fully autonomous contributions” to the most important parts of model training.

Researchers nevertheless described “signs of life” in some areas.

How firm is the 80% estimate?

The 1.9-times estimate is sensitive to method. Different statistical assumptions produced figures from 1.1 to 2.9 times.

The range shows why METR presented a point estimate with both a confidence interval and an uncertainty label. The 80% attribution and the 1.9-times pace are separate findings: one measures the pace of benchmark growth, while the other estimates how much of that acceleration came from AI.

When METR added models from other labs to the trend, measured acceleration fell to about 1.4 times.

METR offered one explanation: Anthropic’s models sat below the industry frontier before late 2025.

The sensitivity check lowered Anthropic’s apparent acceleration relative to a broader industry trend. Because METR offered the frontier position as only one possible reason, the 1.9-times headline did not hold under every specification.

The 1.9-times result also exceeds the 1.7 times in Anthropic’s Haiku 5.5 system card, mainly because METR counted unreleased models.

METR’s earlier research points in both directions. A footnote says time-horizon trends suggested rapid progress, while a 2025 developer study found early-2025 AI tools caused a slowdown.

The report cited both results without declaring the acceleration question settled. Its Anthropic estimate therefore sits alongside METR’s earlier finding that early-2025 tools caused a slowdown.

In October 2026, METR and Redwood Research examined OpenAI’s internal agent monitoring in a separate project. That work did not supply evidence for the Anthropic estimate.

Why does redaction matter?

Much of the evidence behind METR’s conclusions is not public.

Anthropic had the right to strike non-public information from the review.

METR wrote that Anthropic “redacted a majority of the evidence we found most informative.”

Anthropic said its edits were meant to keep out details that could help other AI developers move faster.

It also removed a planned section that would have compared how much compute goes to safety work with how much goes to capabilities. Anthropic told METR it would revisit that analysis later.

Without that section, METR could report confidence that compute growth did not explain the acceleration but could not publish the planned safety-to-capabilities allocation check.

Sam McCandlish, an Anthropic co-founder whose X profile lists him as Responsible Scaling Officer, responded Thursday on X.

He wrote: “We redacted a lot, and I accept that this makes it harder for others to check METR’s work.”

McCandlish said sharing what drives the speedup “risks handing out a roadmap for others to go faster.”

He also said many of METR’s conclusions “match my own.”

The tension is explicit: Anthropic withheld material to avoid giving other developers a path to faster development, while METR said the redaction made its work harder to check.

How did outside researchers react?\