>An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on
Honestly, this confirms some bias I already have from coding — models can get to the best solution pretty fast, but if the solution is something obscure, they usually don’t find it.
From my experience, models that drive optimizers in a loop often get stuck in local minima. ofc, this is just correlation, and not truly indicative, but if you look at the first graph, analyzing precision vs cost, all three model-driven optimizers just bounce back and forth between a precision range, not really expanding the horizon or getting too much worse than the orignal. from our analysis on this data, model-driven optimizers typically don't start from scratch again, they just continue down their optimization path.
It would be interesting to see how a human researcher would do on the same task and same budget. 2 weeks of salary + a Claude Code max plan might outperform all of these auto-optimizers in a loop.
this would be an interesting experiment but i think one of the main benefits of having at least some automated pipeline is that human iteration is quite slow since the information intake speed is a few magnitudes slower. probably some combination would be optimal, like ensuring alignment of objectives at the beginning and tuning along the way
>An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on
Honestly, this confirms some bias I already have from coding — models can get to the best solution pretty fast, but if the solution is something obscure, they usually don’t find it.
From my experience, models that drive optimizers in a loop often get stuck in local minima. ofc, this is just correlation, and not truly indicative, but if you look at the first graph, analyzing precision vs cost, all three model-driven optimizers just bounce back and forth between a precision range, not really expanding the horizon or getting too much worse than the orignal. from our analysis on this data, model-driven optimizers typically don't start from scratch again, they just continue down their optimization path.
It would be interesting to see how a human researcher would do on the same task and same budget. 2 weeks of salary + a Claude Code max plan might outperform all of these auto-optimizers in a loop.
this would be an interesting experiment but i think one of the main benefits of having at least some automated pipeline is that human iteration is quite slow since the information intake speed is a few magnitudes slower. probably some combination would be optimal, like ensuring alignment of objectives at the beginning and tuning along the way
[flagged]