I've been building an experimental reasoning system at Orivael and testing it against ARC-AGI-3.
One of the runs just scored 100% on ft09.
The unusual part:
There is no LLM in the loop.
Not for perception. Not for planning. Not for choosing an action.
The agent reads the raw grid, decides, and acts directly.
Results so far:
• ft09: 6/6 levels, 80 actions, 100.0%
https://arcprize.org/scorecards/9a212601-a12e-4da0-a527-aa69e86bd2b8
• tr87: 4/6 levels, 247 actions, 25.99%
• cd82: 2/6 levels, 21 actions, 8.59%
• bp35: 2/9 levels, 93 actions, 6.67%
• lf52: 2/10 levels, 42 actions, 5.45%
On ft09, the human baseline is 208 actions.
We finish in 80:
ours: 4 / 7 / 14 / 16 / 26 / 13
human baseline: 43 / 12 / 23 / 28 / 65 / 37
Every ft09 level hit ARC-AGI-3's maximum per-level score.
Total model inference cost across these runs:
$0.00
But what surprised me most wasn't the successful game.
It was why the system fails.
Almost every major failure we've seen has been a perfectly reasonable conclusion based on an incorrect representation of the environment.
Examples:
• A sprite sat on a tile using the same color value as a wall, so the system concluded it was surrounded by walls while standing on an empty floor.
• Measurements taken every half-tile aliased. One measurement showed a block while another apparently showed a wall in the same place.
• The agent concluded a move was impossible after testing it multiple ways, except every test accidentally positioned the relevant object one cell outside the useful state.
• A board that appeared complete was actually a scrolling window onto a larger environment.
• Buttons were classified as inert after being tested in one state. They were actually movement controls that only became active after the machine entered another configuration.
The recurring failure pattern is:
Exhaustive over what was sampled gets reported as exhaustive over what exists.
That distinction is becoming much more interesting to me than the benchmark score itself.
And an important caveat:
We absolutely have not solved ARC-AGI-3.
Twenty of the 25 public games are untouched.
In one game we've examined, the system currently can't even identify a legal action.
The interesting divide we're seeing is this:
Once the agent identifies a game's mechanic, it can often become extremely efficient.
The much harder problem is:
How do you recognize what kind of world you've entered without carrying assumptions over from the previous one?
That's what we're working on now.
Official ARC Prize scorecards/replays are in the writeup.
Would particularly love thoughts from people working on ARC, program synthesis, world models, active perception, or non-neural reasoning.
[link] [comments]