I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.
True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?
The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.
What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.
I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.
I wonder if these benchmarks swap words, meaning and more because you might as well be benchmaxxing for specific words. I notice a lot of recurring just structural sentences coming back in smaller LLM models where they're fit for a specific task which is fine because most of the work we do is repetitive and there are patterns to learn but they should be word agnostic which I wonder if LLM can really fix.
None of the tweets, nor the press release, seems to mention how long time it actually took E2E to complete the evaluation, but they do mention it took "12% fewer actions" compared to just Opus 5 without AVO. Feels a bit suspicious they don't break down the timing involved, looking at the diagram from the press release, it gives the impression there is a lot of machinery here, and given they claim fewer actions, each action must be more carefully considered, doesn't it?
Curious to read more about it though, seems the paper for it is here: https://arxiv.org/pdf/2603.24517, I'm not sure I understand if it's better than just Codex with a /goal, as they talk about "can discover performance-critical micro-architectural optimizations" but leave Codex alone for a day or two and you'll get the same results without doing "additional autonomous adaptation" at all.
AVO is also designed to operate across frontier models. While our full public-set result used Claude Opus 5, we additionally paired AVO with GPT-5.6 Sol on a challenging subset of games. In these limited experiments, Sol reached matched levels faster in wall-clock time in several cases, while Opus used fewer environment actions in matched-level comparisons. These preliminary results suggest complementary operating profiles across models, and we leave a broader systematic comparison to future work
I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.
True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?
The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.
What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.
I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.
The 100% score was achieved on the 25 public set, not on the semi-private or private sets.
I wonder if these benchmarks swap words, meaning and more because you might as well be benchmaxxing for specific words. I notice a lot of recurring just structural sentences coming back in smaller LLM models where they're fit for a specific task which is fine because most of the work we do is repetitive and there are patterns to learn but they should be word agnostic which I wonder if LLM can really fix.
Verified high score is just 40%: https://arcprize.org/leaderboard
None of the tweets, nor the press release, seems to mention how long time it actually took E2E to complete the evaluation, but they do mention it took "12% fewer actions" compared to just Opus 5 without AVO. Feels a bit suspicious they don't break down the timing involved, looking at the diagram from the press release, it gives the impression there is a lot of machinery here, and given they claim fewer actions, each action must be more carefully considered, doesn't it?
Curious to read more about it though, seems the paper for it is here: https://arxiv.org/pdf/2603.24517, I'm not sure I understand if it's better than just Codex with a /goal, as they talk about "can discover performance-critical micro-architectural optimizations" but leave Codex alone for a day or two and you'll get the same results without doing "additional autonomous adaptation" at all.
The blog post: https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-...
Using Claude Opus 5, but it can use others:
AVO is also designed to operate across frontier models. While our full public-set result used Claude Opus 5, we additionally paired AVO with GPT-5.6 Sol on a challenging subset of games. In these limited experiments, Sol reached matched levels faster in wall-clock time in several cases, while Opus used fewer environment actions in matched-level comparisons. These preliminary results suggest complementary operating profiles across models, and we leave a broader systematic comparison to future work
Thoughts on Nvidia releasing AVO or even open sourcing it? They've been very open with their models.
AVO: Agentic Variation Operators for Autonomous Evolutionary Search
https://arxiv.org/html/2603.24517v1
Is this AGI? I don't think I can score 100% on ARC AGI.
It is until ARC-AGI-4. Maybe around 73 we will stop.
You'll find that those goalposts are very movable.
Depends on which definition they’ll use today.
Could KITT? Cmdr. Data?
agi with a context of 250k-1mill tokens?
Now we’re talking
The next year is going to be wild folks