The conceptual problem is that we keep wanting to compare AI behavior to that of traditional computers. The proper comparison is comparing AI, and how we trust or delegate to it, to the concept of delegating to other humans or even to domestic animal. Employees can be trained and given very specific skills and guidelines but still have agency and non-deterministic behavior. A seeing eye dog, a pack mule or chariot horse will often, but not necessarily always do what you ask of them. We've only been delegating to deterministic programmable machines for very short part of human history. But ad human societies, we've been collectively delegating a lot of useful activities to non-perfectly-dependable agents (ie each other) for a very long time. As as humans we've gotten done more that a few notable things in the last several millennia with this method. However, humans as delegates or as delegators have also done a lot of horrific things at scale to, both by accident or by design. And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
When compared to how human make a mess of things like in the real world, how high does the bar really need to be for trusting AI agents. Even far shy from perfect, AI could still be a step function improvement over trusting ourselves.
The main difference is that this junior employee can't be held responsible if anything goes wrong. And the company which rented you this employee absolves itself from all responsibility too.
Who do you trust with the keys?
In any well run organization you have multiple layers of controls. The same concept applies here and I think the gp commenter captured it very well.
Yes. I think you can get agents to “Conscious competence” with a lot of well-designed oversight, direction and control. It works, but it’s fragile - nothing like the judgement needed to handle novel situations well.
This feels like the modern incarnation of "packet intent", the mythical security property of knowing what an incoming request is trying to do rather than what it is. Variants of "packet intent" have been sought after going all the way back into the 1980s; it's helpful to recognize the idea when it appears, because it's a reliable marker of what you can't realistically accomplish.
> You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files.
I feel like people may be viewing the past with rose colored glasses. Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash. Most things didn't "just work", but required extensive tweaking to configure your ram, sound card... to work at all.
It's sad, but they should've compulsively hit save after every few letters - it's documented very clearly on page 404 of the manual. It's a real shame that such things couldn't be done automatically until recently, early-2000-era CPUs just weren't sophisticated enough to run advanced, reactive logic like that.
My parents indoctrinated me as a child to constantly hit save because they grew up with that. It was a part of our cultural expectations for "basic life skills to teach children".
This is not just the past. I still have headaches configuring my video card to work with the right CUDA drivers, etc.
The tower of abstractions we're building has reached a height that actually makes everything more fragile, even if the individual pieces are more robust.
Still though -- once you got a workflow, no matter how terrible, it strongly tended to continue to work that way, and it was still much easier to diagnose, fix, and just generally not have unexpected behavior.
This is the issue; agents introduce more unexpected behavior, at least for now.
My gut is that always on "agents who can do things unexpectedly" are a dead-end, but what AI can do is get you to a nice AND predictable "workflow" easier.
e.g. for now I don't like AI for dealing with my info, but I love AI helping me make more and better bash scripts, that deal with my info.
alot of software engineering, especially in complex systems, is still just tweaking retries, alarms, edge cases etc. it might take 3 days to even figure out what went wrong
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
THIS.
I lost so much work in the 90s and 00s. I was a kid, so I had patience and it didn't cost me any money. I can't imagine people losing actual work presentations or projects.
Every piece of software was like this. It was either the app crashing or Windows crashing. I lost Flash projects, websites, PHP code.
Sometimes software would write a blank buffer to file too, so you needed copies.
Version control was one of my favorite discoveries. I clung to SVN for the few years after I found it.
My final major loss was when Open Office on Ubuntu deleted my 30 page undergrad biochem thesis I'd spent a month on. I've never used it since.
Windows 95 Word was also bad. Some poor non-CS student brought his thesis to our computer pool and worked from Floppy with the only copy he had. Panic mode on when the backup file and original did not fit on that floppy any more and Word asked to swap disks for an empty one. We advised him to just continue swapping, eventually Word will have that backup file on the other disk. It worked after an ennerving amount of floppy swaps...
Quality issues are a different vertical within the space of software/user misalignment. The sort of issue the author talks about is more like the malware of the 90-00s era: the software deliberately does something to screw the user.
been building on claude code for a while. the post's framing is right.
mcp gives you open standards on the tool layer but the harness
(claude code, cursor) is still proprietary. your product is one
anthropic decision away from breaking.
the user agent role the post calls for needs open harnesses, not just
open standards. otherwise we end up rebuilding mobile under a new name.
if you've actually migrated an existing claude code setup to one of them, curious how the portability story worked. that's the part i'd been worried about.
The browser analogy holds because publishers wanted browsers. Sites lived with User-Agent and robots.txt because the click paid for it.
AI agents are the destination. No return click to bargain with. That's why Cloudflare just went default-block + 402 Payment Required instead of waiting on a standards body.
Open standards on the agent side are the easy half. Getting sites to show up is the part W3C can't fix alone.
The thing I don’t like about “agents” is that I consider my computer a tool that I use and control. I don’t want it doing things for me: I want to do things through it. I want to be in the driver’s seat. “Notifications” and “Assistants” and now “Agents” break this philosophy. Now there are these things doing “stuff” on my computer for me and I’m just a passenger along for the ride. A computer should be that “bicycle for the mind” as Jobs put it, not some autonomous information-chauffeur, spooning output into my mouth.
i think whats missing is the raison detre of the Agents isnt a new usecase, its a context prune for the same limitations LLMs provide. LLM as Agent is a subset, where the goal of the agent is set by the parent and is suppose to return a pruned context.
if you dont recognize the technical limitations that produced agents youre wearing rose tinted glasses. LLMs arent approaching singularity. theyre topping out in power and agents are an attempt to exentend useful context.
The sigmoid approacheth and anyone of merit should be figuring out how the harness spits out agents, intelligently prunes context then returns the best operational bits, alongside building the garden of tools.
Its like agents are the muscles, the bones are the harness and the brain is the root parent.
The conceptual problem is that we keep wanting to compare AI behavior to that of traditional computers. The proper comparison is comparing AI, and how we trust or delegate to it, to the concept of delegating to other humans or even to domestic animal. Employees can be trained and given very specific skills and guidelines but still have agency and non-deterministic behavior. A seeing eye dog, a pack mule or chariot horse will often, but not necessarily always do what you ask of them. We've only been delegating to deterministic programmable machines for very short part of human history. But ad human societies, we've been collectively delegating a lot of useful activities to non-perfectly-dependable agents (ie each other) for a very long time. As as humans we've gotten done more that a few notable things in the last several millennia with this method. However, humans as delegates or as delegators have also done a lot of horrific things at scale to, both by accident or by design. And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
When compared to how human make a mess of things like in the real world, how high does the bar really need to be for trusting AI agents. Even far shy from perfect, AI could still be a step function improvement over trusting ourselves.
A very talented junior employee that you can't trust with the keys.
The main difference is that this junior employee can't be held responsible if anything goes wrong. And the company which rented you this employee absolves itself from all responsibility too.
Here is a fresh example from today of what junior employee do when given unlimited agentic power : https://www.reddit.com/r/ClaudeAI/comments/1sv7fvc/im_a_nurs...
Who do you trust with the keys? In any well run organization you have multiple layers of controls. The same concept applies here and I think the gp commenter captured it very well.
Yes. I think you can get agents to “Conscious competence” with a lot of well-designed oversight, direction and control. It works, but it’s fragile - nothing like the judgement needed to handle novel situations well.
https://en.wikipedia.org/wiki/Four_stages_of_competence
This feels like the modern incarnation of "packet intent", the mythical security property of knowing what an incoming request is trying to do rather than what it is. Variants of "packet intent" have been sought after going all the way back into the 1980s; it's helpful to recognize the idea when it appears, because it's a reliable marker of what you can't realistically accomplish.
> You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files.
I feel like people may be viewing the past with rose colored glasses. Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash. Most things didn't "just work", but required extensive tweaking to configure your ram, sound card... to work at all.
I remember when the computer crashed and the user hadn't saved recently, we blamed the user.
It's sad, but they should've compulsively hit save after every few letters - it's documented very clearly on page 404 of the manual. It's a real shame that such things couldn't be done automatically until recently, early-2000-era CPUs just weren't sophisticated enough to run advanced, reactive logic like that.
Serializing a document was non-trivial for the first two decades of personal computing. Auto-save would have destroyed performance.
My parents indoctrinated me as a child to constantly hit save because they grew up with that. It was a part of our cultural expectations for "basic life skills to teach children".
This is not just the past. I still have headaches configuring my video card to work with the right CUDA drivers, etc.
The tower of abstractions we're building has reached a height that actually makes everything more fragile, even if the individual pieces are more robust.
We just need one more layer of abstraction to fix that, and everything will be fine
I'm vibe coding this presently. Update soon.
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
That was in the Windows world. Maybe in the Mac world too?
No so much in the *nix world.
Windows seems to have improved its (crash) reliability since then though, which I suppose is nice. :)
Manually editing config files thanks to an obscure thread so that your printer can actually be recognized by the OS
And then having to learn ctrl-q the minute you started working in the shell..
Muscle memory is a bitch!
lol be honest that lunacy was unique to Microsoft, never had to do that with FrameMaker on SunOS
Still though -- once you got a workflow, no matter how terrible, it strongly tended to continue to work that way, and it was still much easier to diagnose, fix, and just generally not have unexpected behavior.
This is the issue; agents introduce more unexpected behavior, at least for now.
My gut is that always on "agents who can do things unexpectedly" are a dead-end, but what AI can do is get you to a nice AND predictable "workflow" easier.
e.g. for now I don't like AI for dealing with my info, but I love AI helping me make more and better bash scripts, that deal with my info.
Wait, I literally still hit Ctrl-S constantly, usually a few times in a row.
Have people outgrown this unnecessary habit? Haha
alot of software engineering, especially in complex systems, is still just tweaking retries, alarms, edge cases etc. it might take 3 days to even figure out what went wrong
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
THIS.
I lost so much work in the 90s and 00s. I was a kid, so I had patience and it didn't cost me any money. I can't imagine people losing actual work presentations or projects.
Every piece of software was like this. It was either the app crashing or Windows crashing. I lost Flash projects, websites, PHP code.
Sometimes software would write a blank buffer to file too, so you needed copies.
Version control was one of my favorite discoveries. I clung to SVN for the few years after I found it.
My final major loss was when Open Office on Ubuntu deleted my 30 page undergrad biochem thesis I'd spent a month on. I've never used it since.
Open Office on Ubuntu 11.10 user here. I can confirm it froze frequently and you would lose everything. it was incredibly frustrating
Windows 95 Word was also bad. Some poor non-CS student brought his thesis to our computer pool and worked from Floppy with the only copy he had. Panic mode on when the backup file and original did not fit on that floppy any more and Word asked to swap disks for an empty one. We advised him to just continue swapping, eventually Word will have that backup file on the other disk. It worked after an ennerving amount of floppy swaps...
Hmm ... no?
I used computers back then and many things just worked fine. I found Windows XP way more predictable and stable than any of its successors.
Quality issues are a different vertical within the space of software/user misalignment. The sort of issue the author talks about is more like the malware of the 90-00s era: the software deliberately does something to screw the user.
been building on claude code for a while. the post's framing is right.
mcp gives you open standards on the tool layer but the harness (claude code, cursor) is still proprietary. your product is one anthropic decision away from breaking.
the user agent role the post calls for needs open harnesses, not just open standards. otherwise we end up rebuilding mobile under a new name.
These are already available. Mistral’s Vibe CLI[1] is open source. Tools like goose[2]are API agnostic.
[1] https://github.com/mistralai/mistral-vibe
[2] https://goose-docs.ai/
thanks, will try both.
if you've actually migrated an existing claude code setup to one of them, curious how the portability story worked. that's the part i'd been worried about.
The browser analogy holds because publishers wanted browsers. Sites lived with User-Agent and robots.txt because the click paid for it.
AI agents are the destination. No return click to bargain with. That's why Cloudflare just went default-block + 402 Payment Required instead of waiting on a standards body.
Open standards on the agent side are the easy half. Getting sites to show up is the part W3C can't fix alone.
The thing I don’t like about “agents” is that I consider my computer a tool that I use and control. I don’t want it doing things for me: I want to do things through it. I want to be in the driver’s seat. “Notifications” and “Assistants” and now “Agents” break this philosophy. Now there are these things doing “stuff” on my computer for me and I’m just a passenger along for the ride. A computer should be that “bicycle for the mind” as Jobs put it, not some autonomous information-chauffeur, spooning output into my mouth.
I like how the author notices that it really got a start with cloud computing.
The most important thing we can do for AI to be a net positive to society is to ensure that its loyalty is to the user, and not the state.
There is no legitimate intermediate position - The skew will go one way or the other.
This is a silly thing to say.
Such a thing can’t be enforced and it can be flipped on a dime.
You should play around with local LLMs and system prompts to experience it.
i think whats missing is the raison detre of the Agents isnt a new usecase, its a context prune for the same limitations LLMs provide. LLM as Agent is a subset, where the goal of the agent is set by the parent and is suppose to return a pruned context.
if you dont recognize the technical limitations that produced agents youre wearing rose tinted glasses. LLMs arent approaching singularity. theyre topping out in power and agents are an attempt to exentend useful context.
The sigmoid approacheth and anyone of merit should be figuring out how the harness spits out agents, intelligently prunes context then returns the best operational bits, alongside building the garden of tools.
Its like agents are the muscles, the bones are the harness and the brain is the root parent.