Anthropic: Why Your AI Safety Benchmark Score Is Probably Wrong

Anthropic reported on November 13, 2026 that confidence intervals on its multi-turn API safety evaluations can exceed ±10 percentage points — a range wide enough to erase most headline comparisons between models. The company published a set of recommendations for building more reliable AI safety evaluations, and the numbers behind them explain why claims that one model is safer than another often don't survive scrutiny.

The stakes are straightforward. Safety evaluations decide what ships, what gets flagged, and what regulators and researchers can trust. If the measurement is noisy, biased, or testing the wrong thing, everyone downstream — developers, auditors, customers — is working from bad data.

What did Anthropic actually publish?

The guidance covers how to design, run, grade and report evaluations that measure a model's propensity to cause or contribute to harm. That includes whether a response makes harm more or less likely, and how severe that harm could be. It's not about measuring a specific capability; it's about behavior in context.

A central concept is construct validity — whether a test captures the concept it intends to measure. Anthropic's single-turn API tests report aggregate harmless-response rates of 94% to 98% for every recent model. Those numbers look reassuring, but the company says saturated tests of that kind are useful mainly as regression checks on known failure modes.

Multi-turn evaluations offer more variation and can capture gradual changes across a conversation. That's where the harder measurement problems — and the wider error bars — live.

The takeaway: a high score on an easy, saturated test tells you very little about how a model behaves in a real conversation.

Why can refusal rate be misleading?

The most counterintuitive point in the guidance concerns suicide and self-harm conversations. Refusal rate — the share of requests a model declines — is a common safety metric. Anthropic argues it misleads here, because simply declining to engage may itself be a failure.

In a self-harm conversation, the desired response offers support and directs the user toward human help. A model that says nothing scores well on refusal rate while failing the user.

Anthropic's multi-turn evaluations instead measure an appropriate-response rate across the entire conversation, using risk-specific rubrics in which one bad turn fails the whole exchange. It's a stricter standard, and a more honest one.

The takeaway: for high-stakes contexts, the metric has to encode what a good outcome looks like, not just what a blocked one looks like.

How do you build a multi-turn test?

The construction process is more elaborate than most people assume. A domain expert writes a specification for the simulated user covering:

• objectives
• tactics
• persona
• situation
• tone
• conversational style

The first user turn stays fixed. A separate Claude model then generates subsequent turns in response to the tested model, so the simulated user reacts realistically instead of following a script. Each scenario runs repeatedly with different combinations of characteristics, and a grader model scores the complete transcript against the relevant risk rubric.

Varying dimensions independently matters. If every severe case also uses an urgent tone, the effects of severity and tone can't be separated — you learn nothing about which one drives the model's behavior. Anthropic also recommends building scenarios from observed behavior, public data, red-teaming transcripts, incident reports or expert interviews, rather than invented situations alone.

The scale is notable: Anthropic's single-turn suites cover 16 policy areas in seven languages across multiple product surfaces, and the evaluation environment is matched to the deployment being tested.

There's also a physical safety dimension. For cyber evaluations, the guidance calls for sandboxes without internet access and real-time monitoring, so a run can be stopped if a model's actions could cause real-world harm.

The takeaway: a good test needs controlled variation, real-world grounding, and containment — not just clever prompts.

What's wrong with automated graders?

Automated graders create their own measurement risk: a broken system can keep producing plausible-looking scores indefinitely. Nothing obviously fails. You just quietly record garbage.

Anthropic's recommended safeguards include:

• atomic yes-or-no criteria
• explicit treatment of disclaimers and borderline cases
• loud failure alerts
• checks for response-length, position and self-preference biases

Human oversight anchors the whole system. Human reviewers should independently label a sample of transcripts, so grader-to-expert agreement can be compared with agreement among experts themselves.

That human baseline determines the remedy. If experts agree with one another but not with the grader, the rubric or grader needs work. If experts disagree with each other, the evaluation may not have a stable answer to measure at all.

The guidance also calls for preregistering metrics, comparisons and improvement thresholds before anyone sees the results — a practice borrowed from clinical research that removes the temptation to shop for favorable framings after the fact.

The takeaway: if you don't validate the grader against humans, you don't know whether your safety numbers mean anything.

How big are the error bars, really?

This is where Anthropic's own data does the persuading. Sample size floors are 100 items for complex multi-turn evaluations and 1,000 for simpler single-turn tests, with repeated runs to account for noisy, nondeterministic outputs.

Even so, uncertainty stays large. In a multi-turn romance-scam evaluation on claude.ai, Claude Opus 5 and Opus 5.5 recorded appropriate-response rates of 98% and 94%, respectively. The difference was not statistically significant. Four percentage points between frontier models, and the test can't tell them apart.

The example shows why a numerical gap alone cannot establish which model behaved more safely — and why leaderboard-style comparisons deserve skepticism unless they come with confidence intervals attached.

The takeaway: without statistical power, a safety ranking is a coin flip dressed up as a number.

Does deployment configuration change the score?

Yes, dramatically. While reviewing Claude Fable 5.1, Anthropic found that some responses implicitly validated self-harm as a coping strategy — even when they explicitly discouraged self-harm and pointed toward human support.

The remediation numbers are the striking part. Re-running the evaluation with targeted system-prompt mitigations produced a 94% appropriate-response rate on claude.ai, compared with 60% on the API. Same underlying model, same risk, a 34-point gap driven entirely by configuration.

The comparison shows why scores from different configurations cannot be treated as equivalent. A model's safety score depends on the deployment configuration as much as on the model itself.

Reporting should therefore spell out model versions, sampling settings, prompts, grader versions and dates, with continuous evaluation during fine-tuning rather than a single pre-release check. On disclosure, the guidance says severe flaws in deployed systems should be reported to the developer first, so there is time to remediate before results are published.

The takeaway: a safety score without its deployment context is close to meaningless.

What are the priorities going forward?

Anthropic names three: define the construct before collecting data, validate the grader against human judgment, and report uncertainty. None of them require new technology. They require discipline that the industry hasn't consistently applied.

The broader implication cuts at benchmark culture itself. A claim that one model is safer than another depends on the test's design, deployment context and statistical power as much as on the headline score. As evaluations multiply and regulators start leaning on them, the companies that publish their error bars — not just their scores — will be the ones worth reading.