The ARC AGI Benchmark: 2 Powerful Tests Exposing AI’s Real Reasoning Gap

A Test Built to Resist Cheating

Most AI benchmarks eventually get memorized. Models train on enough similar data that scoring well stops proving genuine reasoning. The ARC AGI benchmark was built specifically to resist this. Created by François Chollet, creator of Keras, and Mike Knoop, co-founder of Zapier, this benchmark series measures something narrower and harder than most tests attempt. Not what a model knows, but how efficiently it can learn something entirely new.

ARC-AGI-1 launched in 2019. It took five years to meaningfully move the needle. Then came ARC-AGI-2 in 2025, and ARC-AGI-3 in 2026, each one exposing a different weakness in how frontier models actually reason.

The Original Format: Static Puzzles

Every version of the ARC AGI benchmark shares a founding principle. Easy for humans, hard for AI. ARC-AGI-1 established the format. A model sees a handful of input-output grid examples. Each grid uses a small set of colors arranged in a pattern. The model must infer the transformation rule connecting inputs to outputs, then apply that same rule to a brand new test input.

This sounds simple. It is not. Chollet formalized a specific definition of intelligence behind this design in 2019. A system capable of efficiently acquiring new skills and solving novel problems for which it was neither explicitly designed nor trained. By that definition, memorizing training data does not count as intelligence, no matter how impressive the memorized output looks.

Progress was genuinely slow at first. The ARC AGI benchmark remained unbeaten for five years. Then reasoning-focused models arrived. The state-of-the-art score on the private evaluation set jumped from 33 percent to 55.5 percent in a single year, driven by techniques including program synthesis and test-time training. OpenAI’s o3 model became the clearest early signal that a new category of reasoning system had genuinely arrived.

ARC-AGI-2: Raising the Ceiling

Success against ARC-AGI-1 created a problem. Models were starting to clear 75 percent or higher. Released March 24, 2025, ARC-AGI-2 reopened the gap deliberately. It removed brute-force solvable puzzles and introduced harder, more compositional tasks targeting specific weaknesses in symbolic interpretation and contextual reasoning.

The reset was dramatic. OpenAI’s o3-preview-low, which had cleared 75.7 percent on the original ARC AGI benchmark, scored just 4 percent on the new version at launch. Pure language models without explicit reasoning scaffolding scored zero percent entirely. Human testers, by contrast, solved every task in a panel setting, with individual solvers averaging 66 percent.

Progress did resume. The ARC Prize 2025 competition, running from March through November 2025, attracted 1,455 teams submitting over 15,000 entries. The winning score reached 24 percent on the private evaluation set, achieved at a compute cost of just 20 cents per task. By April 2026, the strongest tracked score had climbed to 85, approaching the competition’s grand prize threshold.

The Vulnerability Test-Time Compute Exposed

Here is where the ARC AGI benchmark’s second version revealed something genuinely important, and slightly troubling. As scores climbed through 2025, researchers noticed frontier models approaching 30 percent accuracy through a specific method. Generating thousands of candidate solutions in parallel during inference, then selecting the best result.

This raised an uncomfortable question directly. Do these improvements reflect genuine intelligence gains, or a better ability to exploit benchmark-specific patterns through sheer computational brute force? A model that succeeds by trying thousands of guesses is not necessarily reasoning more effectively. It may simply be spending more compute to paper over the same underlying reasoning gap. This exact vulnerability is what motivated the next, much more dramatic format change.

ARC-AGI-3: Breaking the Format Entirely

Launched March 25, 2026, ARC-AGI-3 abandoned the static puzzle format altogether. This marked the first major structural change to the ARC AGI benchmark since its 2019 debut. Static grids and single-shot answers are gone. In their place sit interactive, turn-based environments the agent must explore in real time.

Each environment presents a 64 by 64 pixel grid using up to 16 possible colors, functioning as a self-contained visual mini-world with its own rules and mechanics. Crucially, the agent receives no prompt, no stated objective, and no instructions of any kind. It must explore the environment, build an internal model of how it behaves, infer an unstated goal, and plan a sequence of actions to achieve it. The benchmark spans 135 original environments, split across 25 public, 55 semi-private, and 55 private tasks specifically to guard against memorization.

The Collapse That Followed

The results at launch were genuinely stark, and they reframed the entire conversation around this ARC AGI benchmark generation. Every frontier model tested scored below 1 percent. Gemini 3.1 Pro managed 0.37 percent. GPT-5.4 Pro reached 0.26 percent. Claude 4.6 scored 0.25 percent. Meanwhile, human testers solved 100 percent of the environments, without particular difficulty according to the ARC Prize Foundation’s own reporting.

This is not a small gap. Models that had been approaching 30 percent on the previous static-puzzle version collapsed to near zero the moment the format shifted to genuine interactive exploration. The implication is significant. Whatever capability those models had developed against ARC-AGI-2 did not transfer to a task requiring real-time adaptation inside an unknown environment. Pattern recognition against static examples and genuine exploratory reasoning appear to be fundamentally different skills, and frontier models remain far stronger at the former than the latter.

A New Way to Score Efficiency

The ARC AGI benchmark’s third version also introduces something genuinely novel in how success itself gets measured. Rather than simply tracking whether an agent eventually solves an environment, ARC-AGI-3 uses an efficiency-based scoring framework grounded directly in human performance data.

This scoring penalizes inefficiency exponentially rather than linearly. An agent that takes ten times more actions than a typical human solver to reach the same goal does not score at 10 percent relative efficiency. It scores at roughly 1 percent. This design choice reflects something Chollet has emphasized directly. Efficiency is a root concept in measuring intelligence itself, not merely a secondary consideration layered on top of raw success rates.

Why Contamination Resistance Matters So Much

A recurring theme across every generation of the ARC AGI benchmark is deliberate resistance to benchmark contamination, the phenomenon where a model appears to reason well simply because it has memorized patterns similar to the test data during training. The 2025 technical report documented this concern explicitly, noting that current frontier reasoning performance remains fundamentally constrained by knowledge coverage, giving rise to new forms of contamination even on supposedly novel tasks.

ARC-AGI-3 addresses this by design rather than by patching after the fact. Its interactive environments cannot realistically be memorized from static training data, since success requires live exploration and adaptation within an environment the model has never encountered, rather than pattern-matching against a fixed input-output pair.

The Prize Structure Driving Progress

The ARC Prize 2026 competition backs this benchmark with genuine financial incentive. It offers 2 million dollars in total prizes, including 700,000 dollars specifically for the first team to reach 100 percent on ARC-AGI-3. Every winning solution must be published as open source, a structural requirement ensuring that whatever techniques eventually crack this benchmark become available to the entire research community rather than remaining proprietary to a single lab.

What This Means for AI Progress

The trajectory across all three versions of the ARC AGI benchmark tells a consistent, sobering story. Each time researchers found a format that models could exploit through pattern matching or brute-force compute rather than genuine reasoning, Chollet’s team changed the format specifically to close that loophole. Static puzzles gave way to harder compositional puzzles. Compositional puzzles gave way to interactive environments requiring genuine real-time exploration.

The near-zero scores on ARC-AGI-3 do not necessarily mean frontier models have made no real progress. They mean a very specific, narrow, but genuinely important capability, adaptive reasoning inside a novel environment with no instructions, remains largely unsolved. This is precisely the kind of fluid intelligence Chollet’s original 2019 definition targeted, and it appears considerably harder to achieve than the benchmark-passing progress on other AI evaluations might suggest.

Conclusion

The ARC AGI benchmark series has done something genuinely rare in AI evaluation. It has stayed ahead of the models it measures for six consecutive years, forcing a fundamental format change every time frontier systems threatened to game it. ARC-AGI-2 reopened a gap that ARC-AGI-1 had nearly closed. ARC-AGI-3 reopened it again, more dramatically, by abandoning static puzzles entirely in favor of real-time interactive reasoning. For now, the honest verdict from this benchmark is unambiguous. Humans solve every environment with ease. The best AI systems in the world cannot yet solve one percent of them.