AGI next year? Yeah, let's just go ahead and call that "optimism".
AGI-to-ASI: This treats it as just a matter of algorithm improvement. I don't think it is. It's at least as much a matter of training data. Once AI reaches human level (if it ever does), then data becomes a serious problem. High schoolers don't get a college education by reading stuff written by high schoolers. They get it by reading stuff written by people who were beyond college. Where's an AGI going to get the written-by-beyond-ASI stuff to train on in order to make it to ASI? It's not.
Alignment: No, controlling the alignment of an ASI is not a solvable problem. It's not even a solvable problem for humans! (See, for example, "treason".) And even if human alignment were a solvable problem, you wouldn't try to control adult alignment with a protocol written by kindergarteners.
So I don't think they can control the alignment of an ASI, even in principle. But that's all right, because I don't think they can build one, either.
I think much of this is extremely wrong.
AGI next year? Yeah, let's just go ahead and call that "optimism".
AGI-to-ASI: This treats it as just a matter of algorithm improvement. I don't think it is. It's at least as much a matter of training data. Once AI reaches human level (if it ever does), then data becomes a serious problem. High schoolers don't get a college education by reading stuff written by high schoolers. They get it by reading stuff written by people who were beyond college. Where's an AGI going to get the written-by-beyond-ASI stuff to train on in order to make it to ASI? It's not.
Alignment: No, controlling the alignment of an ASI is not a solvable problem. It's not even a solvable problem for humans! (See, for example, "treason".) And even if human alignment were a solvable problem, you wouldn't try to control adult alignment with a protocol written by kindergarteners.
So I don't think they can control the alignment of an ASI, even in principle. But that's all right, because I don't think they can build one, either.