GPT-6-Astra solves 49 of 50 physics problems at $2.40 each
GPT-6-Astra matched specialist accuracy on 49 of 50 unpublished theoretical physics calculations at $2.40 per problem, but on the apparent miss the specialists' own reference solutions turned out to be wrong.
Updated
Why it matters
- GPT-6-Astra solved 49 of 50 theoretical physics calculations on TheoPhysEval at specialist-grade accuracy.
- Average cost: $2.40 per problem.
- On the one apparent failure, the specialists' own reference solutions contained errors.
- Researchers said agents lack 'taste' for problem selection.
- The team expects autonomous, top-level AI physics research to arrive soon.
GPT-6-Astra solved 49 of 50 unpublished theoretical physics problems on the TheoPhysEval benchmark at $2.40 per question. The agent matched the accuracy of leading specialists, The Decoder reported this month.
That figure sets a new reference for cost-per-solution in theoretical physics. $2.40 buys a research-grade physics answer, graded against specialist reference work, on a problem that had not been previously published. The agent passed 49 of those comparisons. The 50th exposed the benchmark itself.
What is the benchmark?
TheoPhysEval is a benchmark built around 50 unpublished theoretical physics calculations. The problems had not appeared in journals or textbooks before the evaluation, which prevents the agent from retrieving finished solutions through training data. Each item demands an answer comparable in rigor to what a working theorist would produce, then comparison against reference solutions written by leading specialists.
GPT-6-Astra matched those reference solutions on 49 of 50 items. The benchmark's design choice, unpublished problems, removes the most cited failure mode for large language models on quantitative tasks: pattern-matched recall of a known answer.
What is GPT-6-Astra?
GPT-6-Astra is the large language model agent that ran the benchmark. The Decoder report did not name the operator, the model weights, the deployment configuration, or the host hardware. The benchmark and the cost figure, $2.40 per problem, are the data points the study released. The model name carries the GPT-6 prefix.
What did the run cost?
The Decoder reported an average cost of $2.40 per problem. The figure covers the agent run that produced a graded answer, including API charges and compute. That price converts a research task into a per-call API operation.
The marginal cost of one more attempt at one more problem now sits below the cost of a single cup of coffee. That changes which problems a researcher can afford to test, which ablations a team can afford to run, and which dead ends a project can afford to walk down before abandoning them.
What happened on the one apparent failure?
The 50th problem is where the result turns into a critique of the benchmark, not the model. In cases where GPT-6-Astra's answer looked wrong, the researchers checked the reference solutions written by the specialists. The references contained errors. The Decoder report did not state how many of the apparent failures reversed into agent wins after the corrections; it said the pattern shifted the burden of proof onto the human answer key.
That pattern matters because every benchmark lives or dies by its reference set. Specialist-written reference solutions that contain mistakes reset the comparison: the agent that fails to match them may, in fact, be correct. The TheoPhysEval team drew that conclusion in enough cases to publish the caveat alongside the score.
Why has physics not seen a math-style breakthrough?
The researchers pointed to a gap between physics and mathematics. Autonomous agents have produced headline breakthroughs in mathematical reasoning, including solutions to long-standing open problems. Physics has not matched that record. The researchers named the missing ingredient directly. Agents lack "taste," the intuition that tells a researcher which problems are worth attacking, they said.
The phrase carries weight because it concedes raw capability is no longer the bottleneck. The agent solves the problem. It cannot yet decide which problem to solve. The researchers framed taste as the next frontier rather than a permanent ceiling, and they left open the possibility that learned heuristics could substitute for human intuition in problem selection.
The contrast with mathematics also matters. Mathematical problems can be stated precisely and graded automatically. Physics problems sit closer to interpretation: which approximation to make, which symmetry to exploit, which observable to compute. The researchers' taste remark points at exactly that interpretive layer.
What does "taste" mean here?
The researchers used the word "taste" to name the gap between solving a stated problem and choosing which problem to solve. In their framing, taste covers problem selection, the intuition to read a field and pick the next productive line of attack. Mathematicians have a long-documented record of that selection intuition. Physicists do as well, but the intuition sits closer to approximation choice and symmetry exploitation than to pure logical structure.
The researchers did not propose a measurement instrument for taste. They named the gap and stopped. The implication for benchmark design is that the next round of evaluation will need to score problem selection, not just problem solution. The Decoder report did not announce such a benchmark.
What do the researchers expect next?
The team expects autonomous, top-level AI research to arrive soon. The expectation is grounded in the benchmark result, not speculation. An agent that solves 49 of 50 specialist-grade physics calculations at $2.40 each has cleared the capability bar for routine research calculation. The remaining tasks fall to judgement, supervision and integration with human workflows.
The Decoder report did not name a date for that arrival. It tied the expectation to the benchmark result itself: $2.40 per problem, 49 of 50, specialist accuracy. The researchers read those numbers as the threshold.
What changes for young physicists?
The researchers identified the training pipeline as the first place to feel the result. Early-career physicists learn by working problems the way senior physicists once did, building intuition through repetition. GPT-6-Astra already performs the calculation at $2.40 a piece. The training question becomes what a junior researcher should do that an agent cannot, and how to evaluate that work without falling back on tasks the agent already solves.
The Decoder report did not propose a new curriculum. It framed the issue as a structural shift: a generation of physicists will train alongside agents that handle the same problems at a fraction of the historical cost. The skill set moves from calculation towards question selection, supervision and interpretation, the layers where the researchers said taste still lives.
What changes for equal opportunity?
The $2.40 figure has a direct bearing on who can do theoretical physics. Compute cost per routine calculation has effectively collapsed. The remaining barriers are access to the model, prompt expertise, and the supervision needed to interpret the answers.
Equal opportunity in physics, by this reading, no longer hinges on cluster time or grant budgets for routine calculation. It hinges on who can frame the right question and who can read the answer the agent returns. The researchers listed equal opportunity in the field as one of the two consequences the result already produces.
What does the benchmark not measure?
The TheoPhysEval score reports a single number: 49 of 50. It does not report time-to-solution, confidence calibration, or failure mode distribution. The Decoder did not publish per-problem cost breakdowns, agent reasoning traces, or comparison runs against other models. The single-figure release is consistent with how mathematical-reasoning benchmarks have been reported when the goal is to establish capability rather than to characterize it.
That narrow framing has consequences for any organization that wants to deploy the result. The $2.40 figure is a benchmark number, not a production cost. A team that runs GPT-6-Astra against its own problem set may pay a different price, hit different failure modes, and need different supervision than the benchmark implies.
Where does the field go next?
The TheoPhysEval result moves the conversation from capability to judgement. The agent solves the calculation. The human chooses which calculation is worth solving, checks the result, and decides what it means. Researchers who treat the $2.40 figure as the end state will train physicists for a job that no longer exists. Researchers who treat it as the starting state will train physicists to direct agents, audit their reference solutions, and exercise the taste the researchers say is still missing.
The next benchmark will measure whether the agent can pick the problem worth solving, and the researchers' caveat about taste points to that test.
Original: arxiv.org
More from Rebecca Stone
Show full bio
Correspondent covering companies, deals and strategy in artificial intelligence and machine learning at AI In Context.
291 articles