Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
Well either AA is biased bribed riged whatever or chinese models are benchmaxxed or maybe its both
Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] To what extent Terminal-Bench 4.0 can be trusted, is a separate question, but unlikely to be biased towards one particular group of models: https://preferencemodel.com/blog/terminal-bench-4-is-saturat...
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
Interesting that changes in the index always happen when „non us big tech“ models seem to be a little to good for comfort.
It makes sense for Muse Wonder though as that model does not trade blows with the best frontier models by a longshot.
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
What's the advantage for cognition?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
If you missed the livestream of mimo's training runs, you can literally see the DeepSWE hillclimb on their dashboard.
https://x.com/zhengyaojiang/status/2100316960772800742