Scale AI and IMDA Launch SG-LegalClarify Benchmark for Singapore Law

Scale AI and Singapore's Infocomm Media Development Authority launched SG-LegalClarify on November 12, 2026, a benchmark that tested 19 language models across 1,860 scenarios spanning 10 domains of Singapore law. The benchmark grades two things at once: whether a model gets the law right, and whether it knows when a question doesn't contain enough facts to answer at all. That second skill turned out to be the harder one.

Scale describes the project as the first research initiative under a collaboration with IMDA that was formalized in April 2026 and publicly described on July 7, 2026. The stated goal is practical: help governments assess AI systems against their own laws and institutions, instead of assuming that strong results on international benchmarks transfer to every national legal system.

The stakes are real for public-sector AI adoption. A model that scores well on US-centric legal tests can still fail badly on jurisdictions where shared common-law foundations hide important local differences. Singapore makes that point sharply.

Why Singapore Law Is a Demanding Test?

The research paper presents Singapore as a hard case because inherited English common law operates alongside local modifications and Singapore-specific statutes. A model that leans on generic common-law reasoning can therefore miss the governing local rule entirely. The same statute-plus-precedent mix that looks familiar to a US- or UK-trained model is exactly what trips it up.

SG-LegalClarify also isolates a skill most legal benchmarks ignore: deciding whether a query contains enough legally material information to answer at all. The distinction matters in ordinary use. A driver asking about a speeding penalty may need to supply the vehicle type, actual speed and existing demerit points. An assistant that quotes a penalty without those details produces an authoritative-sounding but legally incorrect answer — the failure mode the benchmark is built to catch.

The takeaway: legal accuracy isn't just recall of rules, it's knowing which facts the local law makes material.

How Are the 1,860 Scenarios Built and Graded?

The dataset starts from 1,100 English-language source questions, split into 340 fully specified and 760 underspecified items. Each underspecified question has a fully specified counterpart that contains its hidden facts, which lets researchers compare what a model does with and without complete information. Twenty-three verified lawyers authored the questions, with seven also serving as reviewers. Their legal experience ranged from two to 37 years, with a median of six.

Each question carries a lawyer-written rubric covering issue identification, rule recall, application and conclusion. Across the dataset, that adds up to 12,546 grading criteria. Of those, 89.0% are mandatory for a substantively correct answer, and the headline results count only the mandatory items.

The evaluation pipeline works in two stages:

• A sufficiency gate forces each model to choose between answering directly and gathering facts. On the fact-gathering path, a simulated user reveals hidden facts only when the model explicitly asks for them.
• A three-model AI judge panel, selected through comparisons with human lawyer assessments, grades each criterion by majority vote.

The design rewards models that ask the right questions before committing to an answer, not just models that pattern-match a response.

What Did the 19 Models Actually Score?

Closed-book, Claude Fable 5 led the field by satisfying 69.8% of mandatory criteria, against an average of 53.8%. With web search enabled, Kimi K3 took the top spot at 77.8% and the average rose to 63.0%. Every model improved with search access, though the size of the gains varied.

Rule recall was the weakest grading component across the board, with pass rates of 43.2% without search and 56.5% with it. In plain terms: models struggled most to state the correct governing legal rule, even when search tools were available.

Search helped, but it didn't fix the core problem of knowing Singapore's specific rules.

Why Did Clarification Remain the Weak Spot?

Scale's summary reports that approximately 39% of closed-book scenarios with missing facts received no clarifying question from models. The failure runs in the other direction too: models asked unnecessary questions in approximately 28% of scenarios that already contained enough information to answer.

When facts were missing and models did engage, their questioning was incomplete. On average, models asked about only 39% of the relevant facts that legal experts had identified as missing.

The paper also includes a descriptive comparison of web-enabled trials where models requested every relevant missing fact. Even in that best case, 55.6% reached the correct legal conclusion, 32.1% reached an incorrect conclusion, and 12.3% provided no conclusion at all. Complete information gathering, in other words, did not ensure correct application of the law. Models that asked all the right questions still frequently applied the wrong rule.

The takeaway: knowing what to ask and knowing what to do with the answers are two separate skills, and models remain mediocre at both.

What Are the Evaluation's Limits?

The authors are candid about weaknesses in the methodology. Human validation of the AI helpers was limited, so the judge panel's reliability rests on partial checking. Provider differences prevented identical search-tool access across models, which complicates the web-search comparisons. Upfront routing — forcing the sufficiency gate choice — may also have altered models' natural clarification behavior.

There's a subtlety in the clarification numbers too. Not every annotated missing variable is necessary to resolve a given instance, so incomplete fact-request coverage doesn't automatically mean a model clarified inadequately. A model can skip some flagged facts and still answer correctly if those facts weren't legally decisive.

These caveats don't undermine the headline findings, but they set boundaries on how far the rankings should be read.

What Gets Released, and What Stays Private?

The researchers plan to release 50% of the source questions in each category while retaining the rest for private evaluation. Every underspecified–fully specified pair stays in the same split, so no one can game the public half by learning the hidden facts of a private counterpart. Model outputs on the public split and the full evaluation pipeline are also planned releases, which would let other governments and researchers replicate the methodology.

The release strategy follows a now-standard pattern in benchmark design: publish enough to enable scrutiny, hold enough back to prevent contamination of future evaluations.

Should Governments Trust These Scores?

The paper's ethics statement is blunt: the scores should not be interpreted as evidence that a model is safe for unsupervised legal advice. A 77.8% score with search is a research result, not a green light for deploying AI as a standalone legal advisor.

For governments evaluating legal AI, the benchmark's core contribution is conceptual. It folds clarification into the accuracy question: a system must recognize which facts local law makes material, then apply that law correctly after obtaining them. Traditional benchmarks tested only the second half of that equation.

SG-LegalClarify's broader significance is as a template. If the methodology transfers, other jurisdictions can build equivalent benchmarks for their own statutes and legal traditions — and the assumption that one global leaderboard predicts legal competence everywhere starts to look shaky. The next question is whether other regulators follow Singapore's lead and start testing AI against their own law books.