Every time I see this kind of story, two things bother me.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.
What I really started to notice is that SOTA models are really good at putting themselves out of the job.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.
I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.
I want small models to win and I am constantly experimenting with them. I have never tried fine-tuning and do not have that kind of budget. My approach is to remove some of the burden from models and bring into the agent.
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
This continuous cycle of fine-tuned open models beating frontier on (often vaguely labeled/defined) benchmarks doesn't provide an accurate comparison to the expanding generalized capabilities of the SoTA, which makes them effectively meaningless.
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
> I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
What if you have to do the task a billion times? Which model will you choose?
This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.
It doesn't mean frontier models aren't good at harder tasks.
Huge SOTA models are like a floodlight. They elucidate a huge area at once.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!
I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
I like the 2x2 grid that describes when to fine-tune a model, when to use a frontier model, etc.
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.
Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.
They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.
Every time I see this kind of story, two things bother me.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.
> algorithms and formulas
But how am I supposed to push my branch without asking fable set to max reasoning? Blasphemy
what are those cheap fine-tuning services these days?
Tinker is pretty cheap. Prime Intellect if you want more flexibility.
the major labs want to advance science. current business use cases are a happy accident
What I really started to notice is that SOTA models are really good at putting themselves out of the job.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart.
Anything else? Sure use a cheaper model.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.
I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.
I want small models to win and I am constantly experimenting with them. I have never tried fine-tuning and do not have that kind of budget. My approach is to remove some of the burden from models and bring into the agent.
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
There seems to be no hold out data for test, so this is just overfitting?
This continuous cycle of fine-tuned open models beating frontier on (often vaguely labeled/defined) benchmarks doesn't provide an accurate comparison to the expanding generalized capabilities of the SoTA, which makes them effectively meaningless.
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
Maybe because people who are target audience don’t need to have it spelled out like that?
People who are not really into it, don’t care.
> I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
What if you have to do the task a billion times? Which model will you choose?
Speed play also you can get much faster responses with 9b model.
This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.
It doesn't mean frontier models aren't good at harder tasks.
That's fair and I agree with this framing.
Huge SOTA models are like a floodlight. They elucidate a huge area at once.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!
I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
I like the 2x2 grid that describes when to fine-tune a model, when to use a frontier model, etc.
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
"87.3%
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.
Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.
The problem i've had with finetuning models is that most of the time better prompting beats finetuning
Better prompting doesn't improve response time or price!
What benchmark is it? Is it super niche?
They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.
Comparing the revenue of the top quartile of AI-using companies to the average of all non-AI-using companies is beyond intellectually dishonest.
Tldr: we don't know what we are doing like the rest of 99% AI teams.