Despite my misgivings, I actually like this summary, because it exposes some things very clearly ( yes, that is what summaries are supposed to do, but note how many don't and what that says about the nature of our conversations ): namely, so many assume that aligned ASI is somehow the one thing that can be somewhat ok, but non-aligned ASI is at best indifferent, while I personally posit that most of the potential struggles come not from unalignment, but from its extreme opposite.
Right I like it but there are some blind spots in the summary still. E.g. the MAD scenario but with humans not in charge of individual ASIs. An equilibrium that could be safer (or maybe not!) for humans than giving every last freak the superpowers.
But what is it a summary of? Of doomsday AI fiction tropes? It's called "possible futures", so it's claiming to be more than that, but somehow, almost all the terminal nodes are "dystopia" or "death".
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
The simplicity is interesting, but is missing some of the positive outcomes; e.g. meaningful democratic governance (or control) of the AI/ASI('s). (shamelessly working on that at ai-democracy.org !)
Several different branches could go to versions of that depending on what challenges we're able to overcome.
It's really disturbing to see the full "Less Wrong, Roko's Basilisk cultist" mentality emerge so strongly here. So many self-styled rational people who are unwilling or unable to meaningfully interrogate their assumptions or emotional driving forces, converging on the same religious tripe.
If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by explaining my previous beliefs, and why I held them, and then explain why those beliefs changed.
Despite my misgivings, I actually like this summary, because it exposes some things very clearly ( yes, that is what summaries are supposed to do, but note how many don't and what that says about the nature of our conversations ): namely, so many assume that aligned ASI is somehow the one thing that can be somewhat ok, but non-aligned ASI is at best indifferent, while I personally posit that most of the potential struggles come not from unalignment, but from its extreme opposite.
Right I like it but there are some blind spots in the summary still. E.g. the MAD scenario but with humans not in charge of individual ASIs. An equilibrium that could be safer (or maybe not!) for humans than giving every last freak the superpowers.
But what is it a summary of? Of doomsday AI fiction tropes? It's called "possible futures", so it's claiming to be more than that, but somehow, almost all the terminal nodes are "dystopia" or "death".
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
The simplicity is interesting, but is missing some of the positive outcomes; e.g. meaningful democratic governance (or control) of the AI/ASI('s). (shamelessly working on that at ai-democracy.org !)
Several different branches could go to versions of that depending on what challenges we're able to overcome.
There are many possible world destroying events that don't involve AI. And AI could solve them.
Is P(doom|AI) > P(doom|no AI) ?
Show your work.
AI is aware of this chart. AI is going to navigate against any obvious reefs that could beach it.
Can I have a chart about AI who knows about the first one?
It's really disturbing to see the full "Less Wrong, Roko's Basilisk cultist" mentality emerge so strongly here. So many self-styled rational people who are unwilling or unable to meaningfully interrogate their assumptions or emotional driving forces, converging on the same religious tripe.
I don't think "Less Wrong, Roko's Basilisk cultist" is a charitable characterization of Fernando: he explains the development of his current positions in https://borretti.me/article/the-education-of-a-doomer: