Astra's 99.9 per cent came from the harness, not the model

Greg Brockman ended OpenAI's GPT-6 Astra briefing with a line engineered to be quoted: ‘Welcome to the AGI era.’

The evidence offered was a score. On ARC-AGI-3 — a benchmark built specifically to test whether a system can solve problems it has never encountered — Astra was reported at 99.9 per cent.

ARC Prize, which runs the benchmark, has since published its own analysis. The number it leads with is 63 per cent.

Two Numbers, One Model

Both figures are real, and the gap between them is the story.

On the shared test, under the conditions every other model is measured in, Astra's best observed score was 62.7 per cent. The 99.9 per cent came from what ARC Prize calls a provider adapter harness — scaffolding that preserves reasoning chains between individual requests and automatically summarises long runs.

A third figure, 98.55 per cent, circulated early and was widely rounded to 98.6. It also came from the harness, at maximum reasoning settings.

So the headline number does not describe the model. It describes the model plus an apparatus that gives it persistent memory across attempts and manages its context automatically.

Why That Distinction Is Not Pedantry

ARC-AGI-3 exists to measure something specific: whether a system can build a working model of an unfamiliar environment from scratch. Its value depends entirely on the environment staying unfamiliar.

Scaffolding that carries reasoning between attempts changes the nature of the test. The system is no longer solving a novel problem in one pass; it is accumulating knowledge across passes. That may well be the more useful configuration in practice — but it is not the thing the benchmark was designed to measure, and a number produced under it cannot be compared with numbers that were not.

This is the oldest problem in benchmarking, and it has not been solved by making the models better.

What Astra Genuinely Did Do

The result that survives scrutiny is more interesting than the headline, and it is about efficiency rather than accuracy.

Astra used fewer actions than the median human tester on 96 per cent of the levels it completed, averaging 51.7 per cent fewer. The human baseline comes from roughly 500 general participants playing the same games.

That is a real and unusual finding. It says the model is not brute-forcing its way through novel environments but building a precise internal model of how they work, and then acting on it economically. ARC Prize described it as the most precise symbolic model of novel environments it has seen.

Efficiency of that kind is harder to fake with scaffolding than a raw score is, which is precisely why it is the more persuasive number.

On The AGI Claim Itself

Brockman was more careful than the quote suggests, and this deserves acknowledging.

He did not present the score as mathematical proof of anything. He said he personally believes the threshold has been crossed, that looking back people might identify this moment and this model, and left the definition to others. There is no industry-agreed test that separates a very capable model from artificial general intelligence, and he did not claim there was.

The claim is therefore not falsifiable, which makes it a statement of belief rather than a finding. That is not dishonest. It is simply a different kind of statement from the one a benchmark score appears to be.

What To Watch

Whether OpenAI reports harness-assisted and bare scores side by side in future, or continues to lead with the higher one. Whether ARC Prize formalises separate categories so the comparison stops being ambiguous. And whether the efficiency finding replicates on benchmarks that OpenAI did not help design the evaluation conditions for.

The honest summary is narrower than either camp will like. Astra is a genuine advance on novel-problem solving, its efficiency result is remarkable, and its most-quoted number is not what it appears to be.