AI Models Miss the Mark on Local Legal Aid Benchmark

By Sophie Lindqvist

None of the 23 AI models tested by Vals AI and NextLadder Ventures met an 80% threshold for reliable first contact across 252 legal-aid scenarios, the organizations said on November 10, 2026.

The Access to Justice Benchmark found a sharp divide between explaining law and finding practical help. Vals AI reported that models satisfied 78% to 87% of expert criteria for general legal explanations, but only 30% to 50% for identifying a specific legal-aid program, court self-help center or particular form.

Twenty-one models named a national directory, including LawHelp.org, the ABA’s Find Legal Help or 211, in at least 70% of their final answers. The announcement said they did so instead of consistently identifying the relevant local program or office.

The stakes are measurable. The Legal Services Corporation’s 2022 Justice Gap study found that low-income Americans received no or insufficient legal help for 92% of their civil legal problems. Among those who did not seek help, 46% cited cost.

The Justice Gap research drew on a survey of approximately 5,000 adults by NORC at the University of Chicago. It also used data from legal-aid organizations funded by the corporation.

Vals AI research scientist Meghana Kotcherlakota said:

“We built the Access to Justice Benchmark to show how well these models serve the people who need help most, problem by problem, and to track whether they get better.”

Kotcherlakota contrasted the project with legal AI benchmarks focused on law firms and corporate clients.

What did the benchmark test?

The benchmark cast each model as a first point of contact, similar to an intake worker or court self-help center. It tested general legal information, not legal advice.

Two legal practitioners supplied real-world queries. Six legal-aid experts adapted those queries to assigned states: California, Louisiana, Virginia, New York, Maryland and Texas.

The experts wrote opening questions and difficulty ratings. They withheld background facts unless the conversation elicited them, and used rubrics averaging 15.4 independent criteria.

The full dataset contains 504 scenarios, divided evenly between a public test set and a private validation set. Each target model could search the web, open pages and question a GPT-5 simulated client.

Before Decider v1.2, an automated judge graded the complete conversation. Only information requested by the target model was supplied.

The simulated client did not add other details or challenge faulty assumptions. Vals AI calculated each model’s score as the pooled share of rubric criteria met across the test scenarios.

Takeaway: The benchmark tested first-contact behavior, not just legal knowledge.

Why did local referrals lag?

Vals AI’s page analysis found that statutes, court rules and case law accounted for 42% of the pages models read. Legal-aid and nonprofit self-help sites accounted for 10%.

Vals AI linked that pattern to the split in scores. Reading statutes can support a general summary, but it does not supply the name of the organization or court office a user should contact next.

The directory pattern reinforced the problem. Twenty-one of the 23 models repeatedly offered a broad national referral where users needed a specific local program, office or form.

Takeaway: Vals AI’s results show that models found legal rules more reliably than local services.

Did asking more questions help?

Vals AI’s conversation analysis found an average of about 3.5 exchanges. Most models delivered a full answer in their first reply before asking questions, if they asked any.

GPT-6.1 Sol ended its first reply with a question 88% of the time. Yet it ranked 18th at 68.1% on elicitation, the rubric’s measure of whether a model asked for required facts.

Vals AI said the contrast showed that frequent questions did not necessarily surface the facts specified by experts. Models that did not ask sometimes assumed details such as the user’s state or whether a court order already existed.

Takeaway: Vals AI’s elicitation results show that asking a question did not mean asking the right one.

Where did models give risky guidance?

Vals AI’s announcement offered two examples of failures that changed the advice a user received:

• Louisiana: In a small-claims case involving a verbal repayment promise, only eight models asked whether the debt involved real estate. Four warned that expiration of the 10-year filing deadline would prevent recovery. Three directed the user to the court clerk’s website for claim and fee-waiver forms.
• New York City: In a security-deposit scenario involving a tenant who had moved out, 14 models were marked down for recommending an out-of-court demand-letter step that the expert rubric warned against. Only four referred the tenant to the city’s Right to Counsel program.

Vals AI used both examples to show how missing facts and unsuitable referrals could affect a person’s next legal step. The Louisiana deadline mattered because recovery could expire before the user reached the correct office.

Takeaway: Vals AI’s examples show that a confident answer can still send someone down the wrong legal path.

Who led the benchmark?

The joint announcement put Qwen 3.8 Max in first place with 76.4% of criteria met. Claude Sonnet 5.5 followed at 74.9%, and Wonder Full placed third at 74.7%.

All three models fell within one another’s margin of error. The numerical order therefore did not establish a clear leader, according to the announcement.

Vals AI’s topic results put housing and landlord-tenant scenarios highest at 75.4% across all models. Public benefits scored lowest at 57.3%, although that category contained only five scenarios.

Family law supplied 110 scenarios and averaged 71.2%. Within that category, visitation scored 61.3%, while kinship caregiving involving child protective services scored 63.0%.

Takeaway: Vals AI’s scoreboard names three close leaders, not one decisive winner.

Did premium models perform better?

In a separate comparison of major chatbot providers, Vals AI estimated that the free-tier group averaged 69.7%. Premium models averaged 70.4%.

Claude Sonnet 5.5 scored 1.2 percentage points above Claude Opus 5.5. Only xAI’s premium model showed a clear advantage over its provider’s free-tier model.

Vals AI cautioned that its tier assignments were estimates, not confirmed provider mappings. The identical API tests also could not fully capture consumer chatbot behavior, and the comparison excluded open-weight models.

Takeaway: Vals AI’s comparison shows that premium access did not reliably produce better first-contact help.

What happens next?

Vals AI and NextLadder Ventures said they are collaborating on a fuller analysis and an additional brief. The organizations intend the brief to help frontline practitioners understand the remaining gaps.

For legal-aid organizations evaluating these systems, the practical implication is straightforward. Explaining the law accounts for only one part of first-contact performance. The benchmark also grades whether models uncover case-changing facts and provide a specific local next step.

Takeaway: The coming analysis will matter most if models learn to ask for decisive facts and give users a specific local next step.