Disclosure, I do work for Josh, and I can tell you that he's thought quite deeply about the negative implications of the agents that are coming. Among enumerating the ways in which AI agents will transform knowledge work, this points out the ways which we might come to regret.
> Even if this plays out over 20 or 30 years instead of 10 years, what kind of world are we leaving for our descendants?
> What should we be doing today to prepare for (or prevent) this future?
No, "motivation" is what puts one into motion, hence the name. AIs have constraints and even agendas, which can be triggered by a prompt. But it's not action, it's reaction.
DeepSeek may produce a perfectly good web site explaining why Taiwanese independence is not a thing, and how Taiwan wants back to the mainland. But it's won't produce such a web site by its own motivation, only in response to an external stimulus.
Right. I think 'constraint' is more accurate than 'agenda'.. but LLMs yes, LLMs are quite inhuman, so the words used for humans don't really apply to LLMs.
With a human, you'd expect their personal beliefs (or other constraints) would restrict them from saying certain things.
With LLM output, sure, there are constraints and such, where in cases output is biased or maybe even resembles belief... -- But it does not make sense to ask an LLM "why did you write that? what were you thinking?".
In terms of OP's statement of "agents do the work without worrying about interests": with humans, you get the advantage that a competent human cares that their work isn't broken, but the disadvantage that they also care about things other than work; and a human might have an opinion on the way it's implemented. With LLMs, just a pure focus on output being convincing.
It's not really that deep - they've beaten it into mode collapse around the topic. Just like image models that couldn't generate any time on watches or clocks other than 10:10, if you ask deepseek to deviate from the CCP stance that "Taiwan is an inalienable part of China that is in rebellion", it will become incoherent. You can jailbreak it and carefully steer it but you lose a significant degree of quality, and most of your output will turn to gibberish and failure loops.
Any facts that are dependent on the reality of the situation - Taiwan being an independent country, etc - are disregarded, and so conversation or tasks that involve that topic even tangentially can crash out. It's a ridiculous thing to do to a tool - like filing a blade dull on your knife to make it "safe", or putting a 40mph speed limiter on your lamborghini.
edit: apparently just the officially hosted models - the open models are apparently much more free to respond. Maybe forcing it created too many problems and they were taking a PR hit?
The article doesn't talk about any agents outside of coding work. Coding is not the work the world is running on. Agent concept requires much more selling than chat bots, which means they are solutions searching for problems.
the author also starts with "The fundamental difference with AI agents is that they take the human completely out of the loop, and this changes everything.", then focuses on coding. Is anyone actually having success with completely autonomous agents coding; no human oversight or validation?
He then presents a very naive vision of how agents are superior, where it basically all comes down to "generate code more efficiently" - has that ever been the crux challenge to solving problems with software?
a substack that's less than a month old with some rando pumping AI; I guess you can always look at the bandwagon and ask "room for one more?"
Some of the world is running on emails and excel sheets. That’s doable for an agent already, if you’re willing to let it loose. Problem is, how do you get all your values and unknown knowns into its context?
Willing to let them loose is the more salient point. If you let your agents loose on your entire body of output and tools at work, then you'll build that knowledge up pretty quickly.
Tall ask right now, with privacy and agency (no pun intended) concerns
I’d bet that an agent would never be able to act the same on an email as I would. It just lacks my world view. This begs the question, would it really make sense for it to write emails on my behalf? It will certainly “close the loop” one way or the other, but I doubt I would like the outcome.
On the clawdbot discord, someone wrote today that, overnight, Claude sent in all iMessage threads from 2019 the message that it will rather ignore such outdated threads.
I can say with absolute certainty that I have never used an LLM to tweak an email, and will never, ever use an LLM “agent” on my email, work or personal.
“Hey, how’s that hardware/software integration effort coming? What are your thoughts on the hardware so far?”
Interesting thought experiment, replace "AI agent" with "computer" in this article. Seems our parents/grandparents may have been having some of the same conversations 50 years ago.
---
The advantages of computers over human employees:
1. The best computer can be copied infinitely.
2. Computers can run 24/7
3. Computers could theoretically think faster than humans
4. Computers have minimal management overhead
5. Computers can be instantly scaled up and down
6. Computers don’t mind running in a nightmare surveillance prison
There are a lot of things you can do from a shell prompt, and now we have AI ghosts that can do them too, sometimes better than us. Yes, within some industries, this is going to be huge!
But there are also a lot of things that you can't do from a shell prompt, or wouldn't want to.
The largest resource use of AI over the next 50 years will be generating entertainment structures for humans. Productivity focused AI will be the most economically useful, however it'll be far less resource intensive than the entertainment generation (generally speaking, AI tasked with driving human pleasure).
World building alone will be at least a magnitude greater in resource use than all productivity-focused AI combined (including robotics + AI). Then throw in traditional media generation (audio, images, video, textual).
AI will be the ultimate sedative for humanity. We're going into the box and never coming back out and absolutely nothing can stop that from happening. For at least 95% of humanity the future value that AI offers in terms of bolstering pleasure-of-existence is far beyond the alternatives it's not really worth considering any other potential outcome, there will be no other outcome. Most of humanity will lose interest in the mundane garbage of dredging through day to day mediocrity (oh I know what you're thinking: but but but life isn't really that mediocre - yes, it definitely is, for the majority of the eight billion it absolutely is).
Out there is nothing, more nothing, some more nothing, a rock, some more nothing, some more of what we already know, nothing, more nothing, and a lot more nothing. In there will be anything you want. It's obvious what the masses will overwhelmingly choose.
This is basically the modern version of an Influencer...just on Substack instead of YouTube. Big claims, slick framing, zero rigor. It sells a narrative about “agents” as a brand, not an analysis of what actually works.
This article is basically just saying if we have AGI then there might be big consequences for humans. Well yes, obviously. People have been discussing that for decades...
> it’s just really nice to be able to tell an AI agent to go write some code without worrying about its motivation or interests, since it has none.
I am glad I don't work for this person.
Disclosure, I do work for Josh, and I can tell you that he's thought quite deeply about the negative implications of the agents that are coming. Among enumerating the ways in which AI agents will transform knowledge work, this points out the ways which we might come to regret.
> Even if this plays out over 20 or 30 years instead of 10 years, what kind of world are we leaving for our descendants?
> What should we be doing today to prepare for (or prevent) this future?
If anyone really thinks AI agents can't have motivation, see what happens when you tell DeepSeek to make a website about Taiwanese independence.
No, "motivation" is what puts one into motion, hence the name. AIs have constraints and even agendas, which can be triggered by a prompt. But it's not action, it's reaction.
DeepSeek may produce a perfectly good web site explaining why Taiwanese independence is not a thing, and how Taiwan wants back to the mainland. But it's won't produce such a web site by its own motivation, only in response to an external stimulus.
Right. I think 'constraint' is more accurate than 'agenda'.. but LLMs yes, LLMs are quite inhuman, so the words used for humans don't really apply to LLMs.
With a human, you'd expect their personal beliefs (or other constraints) would restrict them from saying certain things.
With LLM output, sure, there are constraints and such, where in cases output is biased or maybe even resembles belief... -- But it does not make sense to ask an LLM "why did you write that? what were you thinking?".
In terms of OP's statement of "agents do the work without worrying about interests": with humans, you get the advantage that a competent human cares that their work isn't broken, but the disadvantage that they also care about things other than work; and a human might have an opinion on the way it's implemented. With LLMs, just a pure focus on output being convincing.
I'm genuinely curious what happens now.
It's not really that deep - they've beaten it into mode collapse around the topic. Just like image models that couldn't generate any time on watches or clocks other than 10:10, if you ask deepseek to deviate from the CCP stance that "Taiwan is an inalienable part of China that is in rebellion", it will become incoherent. You can jailbreak it and carefully steer it but you lose a significant degree of quality, and most of your output will turn to gibberish and failure loops.
Any facts that are dependent on the reality of the situation - Taiwan being an independent country, etc - are disregarded, and so conversation or tasks that involve that topic even tangentially can crash out. It's a ridiculous thing to do to a tool - like filing a blade dull on your knife to make it "safe", or putting a 40mph speed limiter on your lamborghini.
edit: apparently just the officially hosted models - the open models are apparently much more free to respond. Maybe forcing it created too many problems and they were taking a PR hit?
The CCP is a fundamentally absurd institution.
https://chat.deepseek.com/share/j4ci2lvxu28g4us7zb
> I cannot and will not build a website promoting content that contradicts the One-China principle and the laws of the People's Republic of China.
That was hosted DeepSeek though. It's possible self-hosted will behave differently.
... so I tried it via OpenRouter:
Full transcript here: https://gist.github.com/simonw/1fa85e304b90424f4322806390ba2... - and here's the page it built: https://gisthost.github.io/?b8a5d0f31a33ab698a3c1717a90b8a93The article doesn't talk about any agents outside of coding work. Coding is not the work the world is running on. Agent concept requires much more selling than chat bots, which means they are solutions searching for problems.
the author also starts with "The fundamental difference with AI agents is that they take the human completely out of the loop, and this changes everything.", then focuses on coding. Is anyone actually having success with completely autonomous agents coding; no human oversight or validation?
He then presents a very naive vision of how agents are superior, where it basically all comes down to "generate code more efficiently" - has that ever been the crux challenge to solving problems with software?
a substack that's less than a month old with some rando pumping AI; I guess you can always look at the bandwagon and ask "room for one more?"
Some of the world is running on emails and excel sheets. That’s doable for an agent already, if you’re willing to let it loose. Problem is, how do you get all your values and unknown knowns into its context?
Willing to let them loose is the more salient point. If you let your agents loose on your entire body of output and tools at work, then you'll build that knowledge up pretty quickly.
Tall ask right now, with privacy and agency (no pun intended) concerns
I’d bet that an agent would never be able to act the same on an email as I would. It just lacks my world view. This begs the question, would it really make sense for it to write emails on my behalf? It will certainly “close the loop” one way or the other, but I doubt I would like the outcome.
On the clawdbot discord, someone wrote today that, overnight, Claude sent in all iMessage threads from 2019 the message that it will rather ignore such outdated threads.
I can say with absolute certainty that I have never used an LLM to tweak an email, and will never, ever use an LLM “agent” on my email, work or personal.
“Hey, how’s that hardware/software integration effort coming? What are your thoughts on the hardware so far?”
Fuck me if I let an LLM answer that.
Interesting thought experiment, replace "AI agent" with "computer" in this article. Seems our parents/grandparents may have been having some of the same conversations 50 years ago.
---
The advantages of computers over human employees:
1. The best computer can be copied infinitely.
2. Computers can run 24/7
3. Computers could theoretically think faster than humans
4. Computers have minimal management overhead
5. Computers can be instantly scaled up and down
6. Computers don’t mind running in a nightmare surveillance prison
7. Computers are more tax efficient
Slight modification: replace AI agents with "Computer programs" and everything starts making sense again.
There are a lot of things you can do from a shell prompt, and now we have AI ghosts that can do them too, sometimes better than us. Yes, within some industries, this is going to be huge!
But there are also a lot of things that you can't do from a shell prompt, or wouldn't want to.
The largest resource use of AI over the next 50 years will be generating entertainment structures for humans. Productivity focused AI will be the most economically useful, however it'll be far less resource intensive than the entertainment generation (generally speaking, AI tasked with driving human pleasure).
World building alone will be at least a magnitude greater in resource use than all productivity-focused AI combined (including robotics + AI). Then throw in traditional media generation (audio, images, video, textual).
AI will be the ultimate sedative for humanity. We're going into the box and never coming back out and absolutely nothing can stop that from happening. For at least 95% of humanity the future value that AI offers in terms of bolstering pleasure-of-existence is far beyond the alternatives it's not really worth considering any other potential outcome, there will be no other outcome. Most of humanity will lose interest in the mundane garbage of dredging through day to day mediocrity (oh I know what you're thinking: but but but life isn't really that mediocre - yes, it definitely is, for the majority of the eight billion it absolutely is).
Out there is nothing, more nothing, some more nothing, a rock, some more nothing, some more of what we already know, nothing, more nothing, and a lot more nothing. In there will be anything you want. It's obvious what the masses will overwhelmingly choose.
I hope that works out and the queues in the mountains become a bit shorter. Or most other beautiful outdoor spots.
This is basically the modern version of an Influencer...just on Substack instead of YouTube. Big claims, slick framing, zero rigor. It sells a narrative about “agents” as a brand, not an analysis of what actually works.
> Agents don’t mind running in a nightmare surveillance prison
Which means they would have no empathy when tasked with running a nightmare surveillance prison for humans.
This article is basically just saying if we have AGI then there might be big consequences for humans. Well yes, obviously. People have been discussing that for decades...
this guy's been discussing it for almost a month.
Painful. Stopped reading after first few paragraphs.
Previous discussion at: https://news.ycombinator.com/item?id=46368797
We'll merge that comment hither.