This is my attempt to explain what the ARC-AGI-3 interactive reasoning benchmark is testing, how the strongest public solutions work, and why I think their dynamic learning loops and harness design matter if you’re building real agents. Coding agents, scientific assistants, legal researchers, browser operators, and robots all run into versions of the same problem: they fail when they can’t inspect the right evidence, remember what happened, test an idea, or improve their method while the task is underway. ARC lets you watch those failures play out inside a tiny game.
I spent several days and a few hundred dollars trying to make frontier AI models solve ARC-AGI-3. On the community leaderboard, runs can cost up to tens of thousands of dollars. The benchmark is a collection of unfamiliar visual games with no instructions. Smaller models often couldn’t form a useful theory of the game at all. Frontier models did better, but even when I gave them the rules, they hallucinated the board, misunderstood the goal, got lost, and forgot things they had already discovered.
The systems reporting perfect scores on the 25 public demonstration games do much more around the model. They provide structured game state, save experience outside the prompt, and let the model use code and search before acting. Some also delegate narrow jobs or save procedures that worked.
The base model still matters a lot. Across several comparisons, upgrading the model or giving it more reasoning time worked more reliably than any one harness trick. I keep coming back to the same question: which parts of today’s elaborate scaffolding will stronger models absorb, and which parts will stick around? My bet is that agents will still need access to evidence, persistent memory, tools, and verification.
ARC-AGI-3 looks like a collection of small, abstract video games. An agent gets a colored grid and a short list of possible controls. It gets no explanation of what the objects mean, what each action does, or how to win.
To make progress, the agent has to act, look at what changed, infer the rules, work out the goal, and carry what it learns into later levels. A bright shape might be the player. A border might be a health bar. Stepping on one tile might change how another object behaves. The game never labels any of it.
Various games and levels in ARC AGI 3
There are 25 known public demonstration environments containing 183 levels. Researchers use them to build, inspect, and compare agents. The official competition tests each submitted agent on unseen games. Teams can’t inspect those games and tune their systems around them, although the agent can learn while it plays. This is meant to separate a general method from a system that became very good at the demos. The Kaggle track also runs offline, so teams can’t call a frontier model API during evaluation. ARC Prize’s competition page explains the different evaluation routes.
Humans don’t arrive at this test as blank slates. A practiced gamer already expects dark regions to be backgrounds, bright moving objects to be players, edge strips to be interface elements, and platformers to progress in familiar directions. ARC removes language and recognizable cultural symbols, but it can’t remove years of visual and game-design experience. The Tufa team talks about these human priors in its Machine Learning Street Talk conversation.
ARC rewards agents that solve unfamiliar games with a small number of actions. That favors efficient adaptation over button mashing, although the score ignores the potentially enormous computation between actions. The technical report explains the metric.
The official launch reported that humans could solve every environment, while the best frontier systems scored < 1% under the standardized evaluation. A few months later, heavily harnessed frontier systems were completing every known public demonstration level.
I could eventually solve every game I tried myself, but it took some effort. I had to poke at the controls, make a few wrong guesses, and gradually figure out what the game wanted.
The evaluation setup matters too. OpenAI found that the same GPT-5.6 Sol model went from 13.3% to 38.3% on the public set after its runner retained reasoning state and replaced rolling deletion with context compaction. The revised setup also used six times fewer output tokens. They made all this clear in their report five days after ARC Prize put Claude Opus 5 at 30.2%.
A standardized runner isolates the model and exposes weaknesses that a polished product can hide. Testing the full product includes its memory, tools, code execution, and context management. Hidden games test whether the whole system can transfer its method.
The public scores show how much performance a minimal runner leaves on the table. Memory, code, search, and verification help the same model stay coherent across a game. We still don’t know whether those methods will generalize to unseen worlds, run cheaply, or count as general intelligence.
As of August 24, several systems had reported scores near 100 on the demonstrations, while cstl led the semi-private Kaggle public leaderboard at 5.99, followed by Tufa Labs at 4.58. François Chollet has stressed that the 25 public games are a demonstration set, not the evaluation set, and that final scoring will use a harder, fully private set. These settings aren’t directly comparable, but the gap is a good reason not to treat saturation of the public games as a solved benchmark.
François Chollet@fchollet
Regular reminder -- the set of public ARC 3 games is called "demonstration set", not "eval set" nor "training set". It is not meant to be used as training data, and it is not meant to be used as an eval. Scores on the public demonstration set are not indicative of scores on the
1:21 PM · Aug 14, 2026 · 32.4K Views
23 Replies · 24 Reposts · 257 Likes
ARC-AGI-3 is still one benchmark. Nobody has shown that its rankings predict success in coding, robotics, science, or business workflows. But it does isolate perception, memory, causal inference, goal discovery, planning, and economical action.
I ran informal experiments on a handful of public demonstration situations using models from OpenAI, Anthropic, and xAI to understand how the agents failed, not to produce a leaderboard score. The runs weren’t controlled: I used external APIs, changed the prompts as I learned, and eventually burned through a few hundred dollars of budget I had set aside.
I tried several versions of the same basic setup:
I gave the model raw number grids, screenshots, and screenshots with grid lines.
I added detailed game rules, recent-action memory, per-level compaction, subgoals, and reflection after a restart.
I built a browser replay that put the exact input, output, executed action, and before-and-after board in one place.
I asked Codex, using GPT-5.6 Sol (Ultra), to build an executable helper for the game.
A few failure modes kept recurring:
The board was easy to misread. A number grid worked better than a plain screenshot, but the models still hallucinated positions and sometimes declared that two visibly different shapes matched.
The agent did not reliably retain consequences. It could walk into a wall, lose stamina, and later try the same move again. It could repeat a rule correctly while failing to apply it to the current frame.
Sometimes Grok stopped playing and asked me what action to take. On another turn, its reasoning summary discussed ACTION2 while its final answer was ACTION3.
My harness created failures too. One model said it could not see the screenshot I thought I had attached. An OpenAI response chain kept the conversation but dropped the system instructions, while my first trace viewer hid the distinction.
The whole project could drift. Codex spent roughly 15 hours building a deterministic solver for one known game. It worked in about 80% of my informal cases, but a fresh Grok run still failed on level one. I eventually hardcoded level one’s moves just to reach level two. That cleared the level and taught the agent nothing.
My first response was usually to add more prompting. Some of it helped, but the prompt gradually became a manual for one game. It did not make the agent reliably notice a failed prediction and carry the lesson forward.
I didn’t come away blaming one vague thing called “reasoning.” A bad move could start with the model, the memory, my harness, or the project design. The strongest systems intervene at different points in that chain.
These ideas didn't come from isolated teams. ARC has an active Discord and GitHub community where independent builders, university researchers, and frontier labs share traces, debate failures, and borrow from one another. I'm naming projects because they offer clear examples, not because any team owns the idea.
The Duck Harness gives the model the current game screen as both an image and Python data. The model can inspect objects, compare frames, crop an area, count cells, calculate a route, or run a search before it commits to an action in the real game.
Its released prompts.py doesn’t sneak in any game solutions. It pushes the model to distinguish interface elements from game objects, compare the correct before-and-after frames, run small experiments while the rules are still fuzzy, and switch to exact computation once it has a decent theory.
The model decides what the scene means and which uncertainty to investigate, while code handles coordinates, counting, search, and bookkeeping.
The ARC grid is a toy version of interfaces we already build for other kinds of agents:
Coding: A few pasted files or a giant repository dump → symbol search, dependency inspection, executable tests, and runtime output.
Scientific research: A prose description of a protein or experiment → structural coordinates, visualizations, numerical tools, raw observations, and provenance.
Legal or patent work: The model’s memory of the law → current statutes, full cases, filings, prior art, citation search, and primary-source verification.
Browser and business agents: A screenshot with no history or application state → structured page data, action logs, reversible controls, and confirmation of what changed.
In the standardized ARC runner, models often get the grid as rows of integers. Humans see a colored spatial display. My experiment with visible grid lines was a tiny version of the same problem.
A long task creates more history than a model can keep top of mind. A bigger context window delays the problem, but the model still has to find the one observation from much earlier that suddenly matters. Eventually, important evidence gets truncated or buried in noise.
Several ARC systems keep a durable archive outside the current conversation. Duke University’s PRO-LONG writes the full text trajectory to an append-only log that the agent can search with code. MIT’s VISTA saves every visual frame, including animations, so the agent can reopen an old image or crop when it becomes relevant later.
MIT VISTA’s persistent memory archive and reasoning traces
The open-source Prime Agent keeps transcripts, files, Python objects, and worker state outside the active prompt.
We don’t hold an entire codebase, case history, or research program in working memory. We use notebooks, source control, databases, and search to recover the evidence we need.
NVIDIA’s NOOA paper offers one comparison: with the same world-model skill, GPT-5.5 scored 50.2 RHAE using structured, searchable memory and 38.4 with ordinary Markdown notes. Both runs used the public demonstration games.
An archive remembers bad theories just as faithfully as good observations. Retrodict checks whether the recorded history already contradicts a proposed rule before the agent spends another game action. VISTA uses a lighter version of the same habit: say what should happen, act, then compare the prediction with what happened.
An agent may preserve a claim, script, or skill long after it has forgotten why it trusted it in the first place. I think of this as understanding debt. The system can still execute the inherited knowledge, but it can no longer explain or audit where that knowledge came from.
A saved lesson should include the observation behind it, where it worked, known contradictions, and a way to revise or retire it.
When an action fails, some agents try the same thing again with a slightly different prompt. ARC rewards a more deliberate loop:
Come up with a working explanation.
Pick an action that separates it from another plausible explanation.
Say what you expect to change.
Compare that prediction with what happened.
Update the explanation before committing to a long plan.
If two theories predict exactly the same result, the action probably wasn’t a very informative experiment. And if the first prediction in a ten-step plan fails, the agent should stop there instead of wasting the other nine steps.
The August 19 arc-skill release turns this into a constraint. Its harness rejects any action that doesn’t include a falsifiable prediction. It also stops a planned sequence as soon as the first prediction fails. Using an unmodified Claude Code with Opus 5, the author reports 100 RHAE across all 183 public demonstration levels in 7,645 actions. The full trace is reproduced in an ARC scorecard.
One thing I noticed in my own runs was that modeling how the game worked was often easier than figuring out what the game wanted.
Once a theory gets specific enough, some systems turn it into code. The agent writes an executable world model that tracks the important state, predicts what each action will change, and checks itself by replaying the history. After the simulator matches that history, a normal search algorithm can test lots of plans without burning actions in the live game.
WorldModeler and Tycho both use versions of this approach. Tycho keeps one actor in charge of the real game, then brings in a specialist to build or repair the simulator only when that extra work appears worthwhile. Sometimes a strong model with a faithful visual archive can solve the game without building a complete simulator at all.
In the August 14 Twin paper, Twin refuses to act until its executable model can replay every transition observed so far. With GPT-5.6 Sol, it scored 93.3 RHAE and cleared 179 of 183 levels. An off-the-shelf harness using the same base model scored 61.1.
Most of those tokens were discounted cache reads. At the short-context list prices available when the paper appeared, Twin’s released counters imply roughly $1.5k - $2.7k before possible long-context pricing and unrecorded cache writes. The run used Codex’s subscription backend, and the authors didn’t publish an invoice, so its actual bill is unknown. Twin minimized game actions while spending heavily on compute.
Building a complete game engine only pays off once the rules are stable and planning is hard enough to justify the work. Before that, flexible language, exact comparisons, and small calculations may be enough.
Many early agents were fixed workflows. A developer picked the prompts, tools, branches, and their order ahead of time. Newer systems such as Claude Code and OpenClaw are more dynamic. The model can inspect a file, run a test, write a script, hand work to another agent, or change direction based on what just happened.
Changing direction is not the same as learning
An agent can take a different path while making the same underlying mistake. A learning agent should be able to improve the notes, tools, procedures, and specialist roles available to its future self.
Tycho hands bounded jobs to a model builder. Prime Agent keeps persistent workers and programmable state. The Continual Harness lets a separate refiner review a trajectory and propose new memories, prompts, skills, or subagent definitions. Outside ARC, Claude Code, OpenClaw, and Hermes Agent all keep some mix of project knowledge, searchable memory, or reusable skills outside the model weights.
A lucky workaround or hallucinated explanation can enter permanent memory as a reusable rule. New procedures need evidence, a defined scope, tests, and a rollback path so mistakes don't become permanent.
In my own experiments, upgrading the model was the most reliable lever I could pull. Tufa came to a similar conclusion when comparing agents. Model choice largely determined which games the system could solve; the harness had a big effect on cost and reliability. Prime Agent’s chart shows the same broad system going from weak results with smaller models to 95.5% with Opus 5.
A July ablation study compared a textual coding agent, executable world models, scheduled simplification, and exact replay verification. Across the matched settings, the most consistent gains came from a stronger model and more reasoning effort. Verification ranked first every time, but it also cost more. Requiring an executable world model didn’t always help, and in some settings the simpler textual agent won.
On August 21, NVIDIA connected AVO, a general coding-agent architecture first used for multi-day GPU-kernel optimization, to the ARC interface. With Opus 5, NVIDIA reports 100 RHAE across all 183 public demonstration levels using 6,624 game actions. AVO keeps persistent memory and has a supervisor step in when the main agent gets stuck. Tycho had already reached 100 on the same demonstration set. AVO reportedly used the same machinery for preserving evidence, responding to feedback, and escaping dead ends in both domains.
NVIDIA says its comparison with VISTA wasn’t controlled, AVO hasn’t been tested on the semi-private or private competition sets, and the announcement omits the ARC run’s token cost and wall-clock time.
Richard Sutton’s Bitter Lesson hangs over these results: general methods that can use more computation tend to win over handcrafted domain knowledge in the long run. Reading it as “use a bigger model” misses part of the point. Search and learning are also general computational methods.
My tentative read is that game-specific color meanings, brittle prompt rituals, handcrafted hints, and one-off solvers will probably disappear as models get better. The more structural parts of a harness look harder to replace: access to external evidence, persistent state, code execution, search, verification, permissions, cost controls, and rollback.
Context compaction, tool use, and learned planning routines may become standard model or API behavior instead of code that each team has to build for itself. Model capability can’t compensate for missing access. A model can’t retrieve a case it was never allowed to search for, inspect a protein structure it was never shown, or know whether a production deployment worked unless it checks the production system.
Running increasingly optimized systems over the same public games won’t tell us which parts of today’s harnesses are patches for 2026 models and which will remain basic agent infrastructure. I’d rather see a general system face an unseen task without developer tuning, build only the temporary scaffolding it needs, and discard or revise that scaffolding when it fails.
In coding, an agent can spend hours working while carrying one bad assumption about the repository. A bigger context window doesn’t necessarily fix that. The agent needs ways to navigate the code, run tests, remember failed approaches, and keep project knowledge that can be corrected later. If it encounters the same type of fix again, it should be able to turn that method into a reusable skill without treating one project’s workaround as a universal rule.
Papers that describe a protein aren’t a substitute for the protein’s structure, and a persuasive explanation isn’t the same thing as a prediction. A scientific agent needs access to raw observations, the right numerical and visual tools, a record of rejected hypotheses, and the provenance behind each conclusion.
A legal or patent agent needs current primary sources and enough search coverage to know what it may have missed. Its parametric memory can’t replace statutes, cases, filings, or prior art. It should retrieve the controlling material, separate authority from analogy, cite the evidence behind its conclusion, and change that conclusion when contrary material turns up.
For business and browser agents, “It eventually finished” isn’t a useful success metric if the system also needed repeated retries, dozens of failed tool calls, human rescue, or made an irreversible mistake along the way. In a business workflow, wasted game actions become latency, compute cost, user interruptions, audit risk, and recovery work. This ARC-AGI-3 analysis from Cognaptus develops that analogy in more detail.
ARC changed what I look for when I test an agent. I want it to notice uncertainty, run an experiment that might prove its theory wrong, keep the raw evidence, and revise its own tools or procedures. I also want it to know when a one-off tool isn’t worth building.
My 15-hour solver managed much of one game. The leading systems can now complete the known demonstration suite, but they often do it at substantial cost and after repeated access to those environments. They still have to show that they can learn this reliably, efficiently, and on first contact with something genuinely new.
If you’re building agents that work across long tasks, I’d love to know how you’re handling this. What do you preserve outside the context window? How does your agent learn from a failed attempt without turning a lucky workaround into a permanent rule?
Reply or leave a comment with what you’ve tried. If you want to follow this rabbit hole with me, subscribe. I’ll keep writing about which of these ideas survive outside ARC-AGI-3 and prove useful in real systems.