2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
look at the bicycle not the pelican. only astra low made a bicycle that resembles a functional bicycle in the low/none category. luna needs xhigh to make a bike. the legs are still wrong, on the wrong side of the bike, in all but like four of the outputs.
>> Are the pelicans still a useful benchmark at all?
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
Are the pelicans still a useful benchmark at all?
I would guess that
1) they're probably in the training data, and
2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
counterpoint https://static.simonwillison.net/static/2026/openai-pelicans...
look at the bicycle not the pelican. only astra low made a bicycle that resembles a functional bicycle in the low/none category. luna needs xhigh to make a bike. the legs are still wrong, on the wrong side of the bike, in all but like four of the outputs.
>> Are the pelicans still a useful benchmark at all?
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
Thus you have your answer.
Oh god the ghibli ai art yellow is infecting the pelicans too now
[flagged]
[flagged]
[flagged]
[flagged]