ARC-AGI-3: Interactive Benchmark for Evaluating Agentic Intelligence
ARC Prize Foundation has released ARC-AGI-3 — the third version of the benchmark for measuring general AI intelligence. Unlike the static tasks in previous iterations, it features 135 interactive environments sized 64×64. The agent interacts with the environment step by step, without hints: it explores, builds a world model, sets goals, and plans actions. The RHAE metric evaluates not just the solution but also efficiency — using a quadratic formula where excessive actions heavily penalize the score. For example, if a human needs 10 steps but the AI takes 100, the result is just 1%.
Calibration was conducted with 486 people in San Francisco: environments are included only if at least two out of ten solve them completely on the first try. Humans achieve 100% success, with a median time of 7.4 minutes.
Top Model Results
On a semi-private dataset, leading LLMs show minimal results without tools:
- Gemini 3.1 Pro Preview: 0.37%
- GPT-5.4 (High): 0.26%
- Opus 4.6 (Max): 0.25%
- Grok-4.20: 0.00%
These models receive a single prompt: "You're playing a game. Your goal is to win." The leaderboard excludes "harnesses" — external programs for solving tasks — to keep the focus on the agent's pure intelligence.
Why Move Away from Static Tasks
Previous versions of ARC-AGI have been compromised: models like Gemini 3 demonstrate knowledge of number-to-color mappings from the benchmark, even though it's not mentioned in the prompt. Data leaked into training sets. The interactive format solves this by requiring online exploration and adaptation.
The benchmark evaluates four key components:
- Exploration: independent study of the environment.
- World model building: understanding mechanics without instructions.
- Goal setting: determining objectives based on observations.
- Planning: optimizing action sequences.
Harnesses vs Pure Intelligence
A separate community track is provided for harnesses. Example: Opus 4.6 with the Duke harness solves one environment at 97.1%, but fails another at 0%. This shows a lack of generalization — the metric measures engineering, not AGI. François Chollet, the benchmark's author and creator of Keras, emphasizes: true AGI handles it without external crutches, like a human in an unfamiliar situation.
Conditions for humans in the test center are similar: no instructions.
Key Takeaways
- ARC-AGI-3 focuses on agentic capabilities, not pattern recognition.
- The RHAE metric quadratically penalizes inefficiency, emphasizing optimization.
- Top models score below 0.4% without harnesses; humans — 100%.
- Data contamination in past versions confirmed by prompt examples.
- Separate track for harnesses lets the community experiment.
The technical report is available for analysis, highlighting the need for new approaches to AI evaluation.
— Editorial Team
No comments yet.