BioBasicsBench: AI agents still get one third of routine lab analysis wrong

The best AI model tested on the new BioBasicsBench benchmark earned 66.7% of available credit on routine wet-lab analysis — meaning it still gets roughly a third of everyday laboratory interpretation wrong. The benchmark, developed with Ligo, a biotech company focused on enzyme design, was released to measure how well AI agents handle the unglamorous, error-prone work that fills a wet-lab scientist's day.

The stakes are practical. Misreading an image or overlooking a problem with the controls scientists use to check an experiment's results can lead to wrong conclusions and wasted weeks of follow-up work. Existing benchmarks lean on charts and images selected for polished scientific papers, which strips out the ambiguity and troubleshooting real experiments demand.

What does BioBasicsBench actually test?

The benchmark contains 100 tasks across five areas of routine wet-lab work:

• protein methods
• imaging and cytometry
• quantification and assays
• nucleic acid methods
• molecular cloning

Tasks range from simple requests, such as reading the molecular weight of a DNA sample in a gel, to detailed assay analysis. All of them involve work the developers say wet-lab scientists should complete reliably.

The inputs deliberately preserve the raw forms of evidence scientists actually confront: gel and blot images, plate photographs, microscopy stacks, sequencing reads, plate-reader exports and even videos of bench work. A single task can combine several data types, and reference sheets such as ladder charts and plasmid maps support interpretation.

Scientists built the tasks using data from their own experiments, pairing each question with expert-written grading criteria. Every task passed an internal review by a Ligo scientist who had not written it, an external review by a PhD-level scientist working in a wet lab, and a final internal review. Reviewers checked the prompt, the experimental evidence, the rubric, author notes and a graded model attempt.

The takeaway: this is a benchmark built on messy, real bench data rather than publication-ready figures.

How are answers scored?

Agents return a structured answer with the requested conclusion, measurement or recommendation, plus supporting reasoning where required. Answers receive credit against individual rubric criteria, which lets the benchmark distinguish a fully correct conclusion from a partially correct analysis. Most tasks test several skills at once: biological knowledge, image and data interpretation, error detection and protocol understanding.

The evaluation covered seven models — GPT-5.2, GPT-5.4, GPT-5.5, GPT-5.6 Sol, GPT-6.1 Sol, GPT-6 Luna and GPT-6 Astra — across 37 model and reasoning-effort settings, with five attempts on each of the 100 tasks per setting. That is 18,500 scored results in total. Scores are reported as mean reward: the average task score expressed as a percentage of available credit, with 95% confidence intervals calculated by resampling whole tasks.

What do the numbers show?

At the 'xhigh' setting, which allows more reasoning per task, GPT-6 Astra earned 66.7% of available credit, against 43.8% for GPT-5.6 Sol — a 22.9-point gap. GPT-6 Astra substantially improves on GPT-5.6 Sol even at low reasoning effort. The developers read this as evidence that stronger native multimodal capabilities, not just more thinking time, may be key to better everyday lab analysis.

The benchmark team highlighted three concrete failure-mode improvements. Astra "demonstrated a better ability to discern the limit of what the data shows and not draw unwarranted conclusions." In one task, GPT-5.6 Sol classified cells as dead based on the signal from a dead-cell dye — but the detector could also pick up light from a second dye, so brightness alone could not justify that call. Astra recognized the ambiguity and called for better controls to separate the signals.

Astra also handles multimodal data better. In a DNA cloning task, GPT-5.6 Sol saw no colonies on the negative control plate and built its conclusion on that observation. Astra noticed the plate was labeled with the wrong antibiotic and correctly rejected it as a reliable control.

And it holds experimental context better. Shown cells drifting in and out of focus across a series of microscope images, GPT-5.6 Sol hunted for a biological explanation for the cells' apparent disappearance. Astra correctly recognized a z-stack: a series of images taken at different focal depths.

Where does GPT-6 Astra still fail?

Higher scores do not yet mean reliable lab judgment. The release documents several cases where Astra's assumptions overrode the evidence in front of it.

In a colony PCR task, Astra correctly distinguished two groups of colony bands but assumed a larger PCR product would identify successful cloning. On that assumption it proposed lanes 4, 6, 7 and 8; the reference interpretation, based on the control comparison, selects lanes 5, 9 and 10. The visual grouping was right, but the experimental identities attached to the groups were reversed — an error that would lead to purification of the incorrect plasmid downstream.

In a microscopy task, Astra concluded that vacuoles are generally darker than the surrounding cell, even though some appeared lighter in the same image. It inspected the lighter regions, noted that contrast varies with focus, then gave greater weight to the cells it judged better focused and dropped the contradictory evidence from its final answer.

Its counting also falters. On one plate, Astra's counting marks failed to resolve close neighbouring colonies, missed visible colonies and marked empty agar, producing an overcount of roughly 12% relative to the benchmark reference. Because the plated volume and dilution were fixed, that overcount inflates the estimate of how efficiently DNA was introduced into cells by the same proportion.

The pattern is consistent: correct intermediate observations do not always guide the final decision.

Why does this benchmark matter?

BioBasicsBench turns the everyday mechanics of biology — interpreting raw data, checking controls, recognizing unsupported conclusions — into a measurable target. Progress here could let scientists spend less time troubleshooting and more time deciding which experiments to run next. The developers also note that tracking these capabilities will help clarify their implications for biosafety and biosecurity, as agents get closer to making autonomous calls on experimental evidence.