Google's Gemini 4 Argon hits 97.5% on ARC-AGI-3 — with an asterisk

By James Calloway

Google DeepMind's Gemini 4 Argon scored 97.5 percent on ARC-AGI-3, making Google only the second lab after OpenAI to cross the 95 percent mark on that benchmark. That single number is the strongest frontier-lab signal Google has produced in months. It is also, on its own, misleading.

The same model scores 49.8 percent on ARC-AGI-3 when run in the benchmark's standard harness (the fixed evaluation setup used for comparable public scoring). The gap between 97.5 and 49.8 is not a rounding error. It is nearly half the test, and it turns on how the evaluation was configured rather than on what the model knows.

ARC-AGI-3 is a reasoning benchmark designed to resist memorization, which is precisely why labs treat it as a proxy for genuine generalization. A 97.5 percent result would place Argon at the top of the public leaderboard conversation. A 49.8 percent result would not.

What does the 97.5 percent number actually measure?

The headline score of 97.5 percent comes from a test configuration outside the standard harness. The Decoder, which first reported the result, notes that the numbers "swing wildly depending on the test setup." In the standard harness, the score drops to just 49.8 percent.

That spread matters because benchmark shopping — reporting the configuration that flatters the model — has become a recurring pattern across the industry. Google is not accused of anything beyond publishing a number with a significant caveat attached. But the caveat is the story. A model that halves its score when the evaluation conditions are standardized has not demonstrated the same robustness as one whose numbers hold steady.

The takeaway: Argon's benchmark peak is real, but the peak and the floor are both Google's numbers, and readers should weight the floor.

Where does Argon fall short?

The Decoder reports that Argon "trails rivals in math and software engineering." Those are two of the most heavily contested evaluation domains in the current model market, and they are the domains enterprise buyers tend to scrutinize first.

The report does not specify which rival models lead, or by how much. That absence is itself informative. A frontier-lab announcement that leads with an abstract-reasoning score and leaves mathematics and coding comparisons vague suggests the gaps there are not small enough to enumerate.

In plain words: Argon appears to be a specialist-level result on one reasoning benchmark, not a across-the-board frontier claim.

Why does this matter for the frontier race?

Google has spent months chasing OpenAI at the top of the model market, and the ARC-AGI-3 result is the first public sign in that period that a Google model is back within striking distance of the frontier. Crossing 95 percent on a benchmark that only one other lab has crossed is a genuine technical marker, whatever the harness caveat.

The competitive stakes are straightforward. Benchmark leadership drives developer mindshare, API revenue, and cloud commitments, and the gap between first and second place on marquee evaluations has been the currency of AI marketing since GPT-4. A credible Google result re-tightens a race that had begun to look like a single-lab runaway.

The caveat, again: one benchmark, one configuration, one model. Frontier claims built on narrower evidence have repeatedly aged badly.

What should buyers and researchers watch next?

The number to watch is not 97.5. It is 49.8, and whether independent evaluators running the standard harness reproduce something closer to one figure or the other. Until third-party results exist, the responsible reading is that Argon is capable of near-ceiling performance under favorable conditions and roughly coin-flip performance under standardized ones.

The Decoder's framing — that Argon is "closer to the AI frontier than any Google model in months" — is carefully worded, and the care is warranted. Closer is not at the frontier, and the unspecified math and software-engineing shortfalls indicate where the distance remains. The next standardized, independent run of ARC-AGI-3 will tell us which of the two numbers Google's model actually deserves.