There's some good ideas and points in here, but this bit threw me:
> # We tried this
> In July 2025 we went full lights-off
Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before.
I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era. TFA calls it out a few sections later with "But surely the models have gotten better since then", but then just writes off any improvement. That does not match my experience.
my perhaps controversial take is that opus 4.1 was smarter than 4.5 for complex engineering work, but 4.5 was faster and "squishier" - it responded better to simpler prompts, it read between the lines of user input better, and that this was really important to converting new users quickly
Opus 4.5 was a massive step change in capability for sure. The opinions of anyone who hasn't bothered experimenting with these things post-Opus-4.5 are not worth a damn.
They still are far from perfect but they're massively helpful.
I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025.
For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long context to me (i don't perceive any intelligence drop at 700-900K tokens). Yeah it's extremely cost-inefficient, but it works.
To me, the thing that stands out about the whole state we're in here is PR review.
Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end.
I'm not sure how the best teams do PR review, from my perspective it sucks. I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed by pulling down the branch and opening the diff with $EDITOR.
These days I think there's really no excuse for the awful UX. Linear (a company that isn't even in the domain of code review) put out a basic PR review feature[0] that is already better than what GH offers. It's simple: point a small model at the PR, group file changes together based on theme, add some commentary and sort by importance (schema changes > openapi spec).
Immediately, so much mental load has been reduced without the reviewer or the requester doing anything. This feature is pretty damn basic, and I think there are obvious next steps like generating visualisations which a dedicated product could find the time to implement.
Keen to hear others thoughts on why this is the wrong approach, or if there are tools in wide use that solve for this, or why this isnt the right problem to focus on.
Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile.
I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling.
We should be reviewing the actual working software. Systems that make it easy and instant to demo any proposed change are what is needed. Code (and specs) are going to fade into obscurity. PR review has already shifted towards validating the product (working software) over the process (code).
The future of software production is more like Replit – not GitHub.
>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile.
This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant
The reason for human code review is:
1. So *you* understand what's going on, not the LLM, and people can ask you questions about it
2. Because LLMs are not that good at code review
It seems weird to brag about literally not doing your job. It sounds like you could be replaced with a python script, what value do you bring?
This is one of the best writeups I've seen of this
a lot of the model's constraints come down to how they are RLed. Discussions online would be a lot better if everyone understood how the labs train the models in a high level (or did a lil data labeling)
> When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part.
The corollary of agents being bad at maintainability but good at coding is that you can vibecode all the parts where maintainability doesn't matter.
So if you build a (domain-specific) modular architecture for your software first you can then just let your software factories loose on building the modules.
Necessarily, better data is what we need, more importantly, better collaboration and better specialization at all. While the title is a bit misleading and clickbait-y, the message is decent.
I disagree with the way that big models are trained on noisy relationships and RL is applied to tone it back down, it represents a stupid amount of compute thrown at this problem at a scale that is often unnecessary.
Its still so early in the game for de facto standards - engineering teams need to experiment and see what works for their own quality metrics not just parrot “standards and methodologies”
This is still the very early days of AI and AI engineering
> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year.
This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/
Seems like it went pretty well if a consulting company is now being started.
I agree with a lot of what Dex Horthy is saying here but on some fronts I feel like he's missing something. Coding well with LLMs, it's not a skill issue, it's an effort/laziness/rigor issue.
In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness. To reiterate, the teams seeing the best results with AI were already high-discipline and high-hygiene.
AI works on data. The better the data, the better the likelihood of a desirable outcome. Code is data. If you have bad code, no matter how awesome the model you let loose on it, you can't get as good a result as if you had good code to start with. This principle has been well known in AI/ML circles since the 20th century.
e.g., if you are doing spec driven development and not seriously investigating formal verification, IMHO you will come up short. Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true.
“Seems like it went pretty well if a consulting company is now being started.”
You interpreted this backwards. Software companies offer consulting when their product cannot stand on its own. See Palantir, Salesforce, etc.
They are successful companies, yes, but not successful products. The product needs to be instantiated and maintained by sales engineers and consultants and customized into something so bespoke that it’s hardly the company’s product anymore.
Some more context on the consulting company: StrongDM was sold earlier this year, about a year after the dark factory was first announced, and the former CTO moved on to this (presumably) in order to continue the idea.
> Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true.
I found this to be exactly right, and in my work I’ve come up with a taxonomy of constraint mechanisms which I keep in mind when guiding agents: generative to constrain the output of the model, interpretive to constrain how the model ‘understands’ code, and elicitative to help it ask the right questions of users.
There's some good ideas and points in here, but this bit threw me:
> # We tried this > In July 2025 we went full lights-off
Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before.
I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era. TFA calls it out a few sections later with "But surely the models have gotten better since then", but then just writes off any improvement. That does not match my experience.
my perhaps controversial take is that opus 4.1 was smarter than 4.5 for complex engineering work, but 4.5 was faster and "squishier" - it responded better to simpler prompts, it read between the lines of user input better, and that this was really important to converting new users quickly
Opus 4.5 was a massive step change in capability for sure. The opinions of anyone who hasn't bothered experimenting with these things post-Opus-4.5 are not worth a damn.
They still are far from perfect but they're massively helpful.
I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025.
For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long context to me (i don't perceive any intelligence drop at 700-900K tokens). Yeah it's extremely cost-inefficient, but it works.
To me, the thing that stands out about the whole state we're in here is PR review.
Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end.
I'm not sure how the best teams do PR review, from my perspective it sucks. I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed by pulling down the branch and opening the diff with $EDITOR.
These days I think there's really no excuse for the awful UX. Linear (a company that isn't even in the domain of code review) put out a basic PR review feature[0] that is already better than what GH offers. It's simple: point a small model at the PR, group file changes together based on theme, add some commentary and sort by importance (schema changes > openapi spec).
Immediately, so much mental load has been reduced without the reviewer or the requester doing anything. This feature is pretty damn basic, and I think there are obvious next steps like generating visualisations which a dedicated product could find the time to implement.
Keen to hear others thoughts on why this is the wrong approach, or if there are tools in wide use that solve for this, or why this isnt the right problem to focus on.
0 - https://linear.app/docs/diffs#guides
Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile.
I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling.
We should be reviewing the actual working software. Systems that make it easy and instant to demo any proposed change are what is needed. Code (and specs) are going to fade into obscurity. PR review has already shifted towards validating the product (working software) over the process (code).
The future of software production is more like Replit – not GitHub.
>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile.
This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant
The reason for human code review is:
1. So *you* understand what's going on, not the LLM, and people can ask you questions about it
2. Because LLMs are not that good at code review
It seems weird to brag about literally not doing your job. It sounds like you could be replaced with a python script, what value do you bring?
What's your job as an "engineer" in this post-automated world?
QA?
PRs sucked to review long before Agents were a thing, but now it really sucks because there are more to review.
> group file changes together based on theme, add some commentary
Isn't that what commits are? Or ... should be?
I think the right shape is to review and merge directly from the agent window.
This is one of the best writeups I've seen of this
a lot of the model's constraints come down to how they are RLed. Discussions online would be a lot better if everyone understood how the labs train the models in a high level (or did a lil data labeling)
> When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part.
The corollary of agents being bad at maintainability but good at coding is that you can vibecode all the parts where maintainability doesn't matter.
So if you build a (domain-specific) modular architecture for your software first you can then just let your software factories loose on building the modules.
Necessarily, better data is what we need, more importantly, better collaboration and better specialization at all. While the title is a bit misleading and clickbait-y, the message is decent.
I disagree with the way that big models are trained on noisy relationships and RL is applied to tone it back down, it represents a stupid amount of compute thrown at this problem at a scale that is often unnecessary.
The rest of it is on point.
Wait these arent “software factories” they are strung together ai rube goldburg machines
Its crazy to me people write these articles and create standards like this is some kind of engineering standard with years of research and experience
This is like calling these folks the experts on aviation: https://youtu.be/M9Yww9LG3gw?is=xgtA-xMpNy-09Asu
Its still so early in the game for de facto standards - engineering teams need to experiment and see what works for their own quality metrics not just parrot “standards and methodologies”
This is still the very early days of AI and AI engineering
> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year.
This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/
Seems like it went pretty well if a consulting company is now being started.
I agree with a lot of what Dex Horthy is saying here but on some fronts I feel like he's missing something. Coding well with LLMs, it's not a skill issue, it's an effort/laziness/rigor issue.
In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness. To reiterate, the teams seeing the best results with AI were already high-discipline and high-hygiene.
AI works on data. The better the data, the better the likelihood of a desirable outcome. Code is data. If you have bad code, no matter how awesome the model you let loose on it, you can't get as good a result as if you had good code to start with. This principle has been well known in AI/ML circles since the 20th century.
e.g., if you are doing spec driven development and not seriously investigating formal verification, IMHO you will come up short. Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true.
“Seems like it went pretty well if a consulting company is now being started.”
You interpreted this backwards. Software companies offer consulting when their product cannot stand on its own. See Palantir, Salesforce, etc.
They are successful companies, yes, but not successful products. The product needs to be instantiated and maintained by sales engineers and consultants and customized into something so bespoke that it’s hardly the company’s product anymore.
Some more context on the consulting company: StrongDM was sold earlier this year, about a year after the dark factory was first announced, and the former CTO moved on to this (presumably) in order to continue the idea.
Disclaimer: I'm a former StrongDM employee
> Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true.
I found this to be exactly right, and in my work I’ve come up with a taxonomy of constraint mechanisms which I keep in mind when guiding agents: generative to constrain the output of the model, interpretive to constrain how the model ‘understands’ code, and elicitative to help it ask the right questions of users.
Full write up is here: https://www.research.autodesk.com/blog/constrain-agent-not-u...
Dex you aren't part of the slop cannon, you _are_ the slop cannon
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
(and please particularly avoid personal attacks on this site)