What Comes After AGI?

(institute.deepmind.com)

87 points | by jdkee 10 hours ago ago

171 comments

  • GalahiSimtam 4 hours ago

    I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI".

    You might find more people believing that AGI is already solved. For example, the author of "The AGI chronicles" drops this in the epilogue. For him, AGI is achieved, because it can help him with a wide range of subjects, with the ability of a random stranger. It's clear that he means chatbots, internet surveys summaries, and coding bots - he does not even mention self-driving cars there.

    This is a shift from the earlier chapters, where AGI term is employed somewhat more vaguely, and even quipping once that AI researchers think of AGI as a slightly smarter version of themselves.

    I too, have a lower bar for AGI. It would be enough for me if it could design games I work on. Merely speeding up implementation tasks is not that interesting. In the immortal words, a delayed game is eventually good, but a rushed game is forever bad.

    • jaapz 3 hours ago

      > but a rushed game is forever bad.

      Is it though? CDPR famously released Cyberpunk rushed and in a miserable state. But the game is amazing now, and this is a widely held opinion too.

      • dracotomes 2 hours ago

        I played Cyberpunk on release and replayed like a good half of it last year. I see this sentiment a lot online but it doesn't resonate. For me it still feels like the same empty beautiful world populated with same 50-70 NPCs that spawn in just around the corner. Driving around listening to the radio was and is the most fun part about it.

        • raffael_de an hour ago

          actually same with all GTA6 clips I have seen that supposedly show how amazing natural this simulation feels ... which it doesn't at all whatsoever afaic. it's a cringe festival in uncanny valley.

      • bakugo an hour ago

        No, the game is not amazing now. It's not as bad as it was on release, but it's also still nowhere near what was promised before release.

        This only seems like a widely held opinion due to the extremely aggressive post-launch marketing campaign from CDPR, particularly centered around the anime.

      • EtienneK 3 hours ago

        Same with No Man’s Sky.

        • konart an hour ago

          Sure, what's the point for me? I've bought it when I was interested in the project and had time.

          Since then this is just a 50 bucks I've burned. Is it good? Maybe, I'll never know.

      • fooker 2 hours ago

        The game being amazing now is the best propaganda I have seen from the gaming industry.

        No, they just fixed the surface level inconsistencies and issues.

        Free roam is wildly boring, every car handles similarly, you are somewhat invisible to NPCs in general. You can go into a handful of indoor spaces.

        I’m sure they’ll fix all these in the next game and call it the best game ever, while the first one promised most of these before launch.

    • paxy an hour ago

      My personal definition is:

      AGI = as good as most humans at most tasks

      ASI = significantly better than all humans at all tasks

      • HarHarVeryFunny 3 minutes ago

        Those definitions seem to be ignoring the "G" - general(ity).

        I guess when the term AGI was coined, with the "general" being the whole point, to differentiate it from narrow intelligence(s) such as chess-playing ability, that no-one was really anticipating the LLM type of AI - basically a forest of narrow intelligences, aka "jagged" intelligence.

        Of course there is no one agreed upon definition of AGI that people want to use - it was always more of a future aspirational sort of thing, along the lines of "human intelligence", rather than a concrete technical list of capabilities to be nailed down.

        Some of what's missing from LLMs aside from generality, but closely related to it, are things like continual learning (not continual memorization) and traits like curiosity, boredom - the capabilities/modalities that kick in in unfamiliar situations allowing and encouraging us to learn and to be creative, to move beyond a brittle forest of isolated competencies to an arbitrarily explored landscape - generality.

        Working with LLMs certainly has its own feel due to their essentially frozen-at-training-time, and things like their emotionless "you're right to push back" responses every time you hit one of their knowledge gaps and they are 100% wrong.

        It seems unavoidable that people will choose to call some level of LLM intelligence, whether already achieved or not, "AGI", but given its peculiar nature a more specific name such as ALI (artificial language intelligence) would really be more accurate, reserving the label AGI for something more animal like that learns for itself (thereby achieving generality) by interaction with the world.

      • netdevphoenix an hour ago

        Then, arguably, we reached AGI around 2024 or so. The common denominator of most humans at most tasks, has you, assuming that by "most humans" you mean "most living adult humans", set a fairly low bar. Arithmetic, basic literacy skills, ability to follow basic commands, limited ability for learning basic concepts and little else. Coding agents are way above your definition of AGI.

        imo AGI should only include humans with graduate level skillsets.

        • adrianN an hour ago

          Definitions of general intelligence that make the average human not generally intelligent are somewhat problematic.

        • kgeist 43 minutes ago

          In 2024, most models were pretty bad at long horizon agentic tasks. I'd say it's mid/late 2025

        • rimeice an hour ago

          > most tasks > Coding agents are way above your definition of AGI

          …apart from 99% of other tasks humans do that aren’t coding.

    • spacedcowboy 2 hours ago

      > I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI"

      Me too, my first neural network research was for my PhD, 35 years or so ago, on a Meiko computing surface.

      My bar for AGI over what we have (generative AI or genAI) is still fairly high, fortunately I do not believe that the current genAI path is set to lead us to AGI, there are very fundamental differences.

      Right now for genAI, the model is static, the weights do not change, all “memory” is in the context, and even with the context sometimes exhibiting “J-space” (a shared "workspace" for deliberate, multi-step reasoning spontaneously created in the context memory), its ephemeral; it doesn't inform the weights in the base model. Even when we try to do “weight-space programming” to actually modify the model itself, the supervisor-model needed to do that is far and away larger than the one being operated on, and once a change has been made the supervisor-model is now out of sync with its original representation and it becomes less useful the more it diverges. We have not solved "thinking" today, we “just” have a high-probability path between states in a static weight-space which we follow and feed back.

      What thinking requires is the ability to generate hypotheses (which requires internal state modification), to hold them and test them against other facts and hypotheses, and to update the internal state on test results, negative or positive. LLM’s don’t do that, they’re actually more like a database lookup, where the input tokens act as keys into a high-dimensional space where information is stored. You could even consider “the model” to be just an entropic compression of lots of data, and the inference step to be the lookup. It’s the scope, and the fuzziness of the lookup that makes them seem to work so well. Lots of results come back, and the best-path through those results wins. It’s not “thinking” as we understand it.

      This is why humans are still a crucial component: we set the boundary conditions of these lookups, we set the starting point and the goals, and we adjust the success criteria to guide the chat - note that these are all the comprehension parts of the task. It’s also why models will hallucinate: the fundamental design does not lend itself to a null hypothesis. Even in the newest "reasoning" models, there is still no feedback of state back into the model itself. What they are effectively doing is running a Monte Carlo pass through the problem space—generating hidden branches and picking a likely path—before reporting back. Because the underlying weights never update from that process, it’s still just a sophisticated search, not genuine comprehension - there is still no “that makes no sense, reconfigure” step.

      Stuart Kauffman (“The Origins Of Order”) would say its weights are stuck in the frozen realm of order, and that it takes a genuine phase transition to get to where the real action is - at the boundary of chaos and order where humans reside. Tweaking and fiddling with the system while it remains locked in the stasis of static weights is never going to effect that state change.

      That’s not to say genAI is not useful, it’s freaking awesome, possibly the best invention in the history of time, but it doesn’t truly “think”, it maps.

      • kenjackson 25 minutes ago

        It seems like your definition of AGI is too focused on implementation details. IMO it has to be focused on behavior. Once you say something like “the weights do not change” then I think we’ve lost the plot. Instead say this is what an intelligent being can do, but a GenAI cannot.

      • curt15 an hour ago

        Self-modifying neural nets?

    • zobzu 2 hours ago

      it's just marketing. the asi will be solved by some other acronym won't. and neither will be what they say they sell: actual sentience

    • juped 3 hours ago

      Artificial "General" Intelligence has to be general; this term is from the era when the frontier of AI was "good, specifically, at playing Go" or something. Any robot apocalypse connotations are limited to the robot apocalypse fandom.

      • pixelpoet 3 hours ago

        Humanity is in for a very rude awakening if they were to apply their lofty ambitions and forever-moving goalposts about AGI to themselves.

        A solid 50% of Reddit posters are completely defeated by the deep mysteries of they're vs their, nevermind solving Navier-Stokes or playing games etc. I doubt the situation is more hopeful on Instagram or tiktok.

        It's all Kim Kardashian shit with huge rubbery butts and lips, no one is even playing the thinking game. We reached Idiocracy (before you dismiss this as hyperbole, watch the full White House UFC event) well before the machines reached AGI.

        • EliRivers 3 hours ago

          I am unconvinced not only that humans have general intelligence, but also that we would recognise general intelligence if we came across it.

          • hagbarth 3 hours ago

            Humans individually, no. Humanity as a super-organism, much closer.

      • cicko 3 hours ago

        But it's still not good at playing Go. I mean, the one that wins Go tournaments is specialized in Go. And specialized is the opposite of "general". General intelligence is just nowadays getting better at chess. So, it will be some time before it is good at Go. *And* everything else. Not "just Go".

  • jonathanlydall 3 hours ago

    I was recently thinking of the depiction of AI in Star Trek. Specifically TNG, DS9 and VOY, and specifically their "Computer, ..." prompts, not the more advanced kinds like for Data or holograms (including the EMH).

    I recall thinking at the time I first watched them that it was a bit weird that the computer seemed capable of answering general questions and doing things like data analysis, but that seemed to be all it was ever used to do.

    And I think this is exactly the kind of AI we now have today, you can ask it a general question which it can somewhat mostly answer correctly, but requires human guidance to be applied effectively at real world problems.

    • francisofascii a minute ago

      [delayed]

    • FinnKuhn an hour ago

      I always imagined AI as what you saw in futuristic movies. There it was generally text or language based. We generally have that now.

      What we don't really have is physical AI, e.g. autonomous robots, human looking robots, self-driving cars, etc. I think this is what is next.

      • XenophileJKO an hour ago

        Why do people keep saying self driving cars? I'm surrounded by them every day.. like I see probably 10-20 on the way to work.

        • slfnflctd 7 minutes ago

          Probably because the rollout was far slower and more expensive than many were led to expect it would be, is still not profitable, and still requires new custom software changes for every additional city it operates in rather than being able to just drive anywhere.

    • aeonik 2 hours ago

      I had the same thought.

      When when I got my CS degree, I revisited this thought, and reinforced it.

      I thought, this is not how AI and Statistics work. We would need to have an ensemble of thousands of specialized models to run a ship.

    • threethirtytwo 2 hours ago

      The ai we have today is even stranger than the star trek computer.

      On one hand it is far superior to humans. On the other hand it needs our guidance. Both of these are true at the same time.

  • jeanpah 9 hours ago

    We are treating AGI like is something that we can measure. We don't even know what intelligence really is let alone measure it.

    Maybe we can get something that resembles intelligence but in reality is another thing. Maybe intelligence is emergent but not the only thing that can emerge, maybe there are things that share some characteristics with the intelligence we know and aren't intelligence.

    • hologramofham 5 hours ago

      I'm not even sure intelligence is the issue. Is something that looks and behaves like intelligence intelligent? Sure, why not? What's the big deal?

      I think what's really underlying the "fear of calling something intelligent" we have, is more about who we are as a species.

      We historically used intelligence as a yardstick for racial and species superiority. We're always judging whether something or someone deserves respect, deserves consideration. We judge and decide who is superior.

      We do it amongst ourselves so of we are projecting that shit onto AI now. And we are afraid of anything that threatens the superiority complex we have as a species.

      Why? Because our behaviours (call it intelligence, whatever) is how we survived, how we "won" in the fight against nature. We're at the top now. It's safe here.

      So anything that threatens that position is going to be called an enemy because we're still a scared creature jumping at shadows in caves. Can't escape that, it's where we are from.

      That's why we are both so fascinated by, and mortally afraid of, something else having "intelligence". It's our only weapon, our only defence.

      The TV show "Humans" from ages ago made a good story about what AI brings out in us as a species. We both crave connection with something, but at the same time are horrified and incredibly afraid.

      Anyway my point is I think we do know what intelligence is, if we just start being brutally honest with ourselves. It's not that complicated really.

      • pluralmonad 3 minutes ago

        > We're at the top now. It's safe here.

        On the contrary, we have rarely been in so much existential danger as we are now. There was that one ice age that nearly got us, but we are messing with the foundations of our ecology right now. When the bugs go, we go soon after.

      • willy_k 3 hours ago

        Gosh how did people completely forget about soul so quickly. Scientism (material-first dont-recognize-anything-without-logical-or-emperical-basis) has robbed a generation of intelligentsia of honest discourse over humanity.

        • XorNot 3 hours ago

          How?

          If you can't measure or observe it then you're just making it up and at that point everything's valid and right provided it's what I believe. And if you disagree with me then clearly you are not adequately human and we will need a crusade to rid the world of your kind.

      • discreteevent 3 hours ago

        Why not call it machine learning? Some people have many different words for snow. That's because snow can have different qualities and the people who came up with those names are intelligent and want to be able to communicate effectively.

        Machine learning definitely has a different quality than human intelligence. If someone says "machine learning" I know what they mean and what to expect. So why not have at least one different term to describe it? - It's because people want to give it attributes that it does not have.

      • willis936 3 hours ago

        If we are projecting the idea of intelligence superiority onto something with more intelligence wouldn't we expect a creation of superior intelligence to treat us how we treat less intelligent creatures? It seems like the fear is more rational than you make it out to be.

        • cowlevel 3 hours ago

          maybe we shouldn't create those creatures?

    • torben-friis 2 hours ago

      >We are treating AGI like is something that we can measure. We don't even know what intelligence really is let alone measure it.

      Here's a nice test: Can an AI fully substitute you?

      I don't mean replacing you at your job. I mean, if you take a human body and route its higher brain function to an AI, would it be able to perform convincingly?

      I would say currently we're very far from such a scenario.

      When I consider the option to buy a shirt, the problem is considered in a large amount of dimensions intuitively (is it warm enough, does it fit, does it clash with my wardrobe, will my girlfriend like it, will my friends laugh at me for wearing it, does it make me look older, is it a good price, are sales happening soon, is it annoying to clean, can I afford it, will carrying a bag with the purchase annoy me for the rest of the day).

      Currently AI can offer good answers for any of those dimensions if explicitly asked, but it is pretty bad at the "intuitive creativity" of surfacing those points of consideration when relevant. It will at most chose a couple of criteria and fixate on them.

      I think this ability, however it's defined, is a missing prerequisite for AGI. Without it, you can have something that outperforms humans at every individual task and still sucks in real world decision making.

      • Jtarii 2 hours ago

        I don't think intuition about the things that a specific type of monkey cares about is a prerequisite for "AGI".

        • thmoonbus 10 minutes ago

          I mean we did create the technology and came up with the concept of intelligence, as we understand it (and defined the term AGI based on our concept of intelligence).

          Why do people insist on talking about AI like it’s some supernatural, otherworldly creation? No matter how advanced it is, it’s still human technology.

        • danlitt 2 hours ago

          "a specific type of monkey" is a funny way of saying "the only generally intelligent thing we are aware of".

    • ionwake 4 hours ago

      People need to start measuring threats by skill level. Agi is already superior to me at code and architecture. Its skill set surpasses me. It doesn’t do everything I do but its skill set includes the hardest stuff.

      A 12 year girl from a third world country with little education and a normal cloud ai would far surpass me at work if I did not have access to AI, yet comes close if we both had it. AGI already passed a few months ago. SI or happy ai or whatever the us wants to call it may or may not come but if it does it’s skill set may drown out my understanding of what is even occurring.

    • bigbaguette 9 hours ago

      100%. And It’s pure hubris to pretend that “intelligence”, especially of the general kind will be reached with language alone.

      • origini 4 hours ago

        I can explain an issue to some AI and it will 'understand' faster and better than human co-worker.

        AI is here doing literaly the same job as a human who needs to be intelligent.

        I see your stand as hubris from your side.

        For such a strong viewpoint, you deliver not any context of why you think like this.

        • layer8 3 hours ago

          Why did you put “understand” in quotes?

          • origini an hour ago

            Because I don't think we are at the point that the word understanding matches what AI does.

            I would say yes it understands things but it was a causios 'understand'

      • semiquaver 8 hours ago

        Maybe you’re right. But it’s also wise to be cautious, just in case we do accidentally create something more intelligent and potentially more powerful than humanity.

        You have to admit the prospect looks more likely than ten years ago.

        • Forgeties79 8 hours ago

          We’re also closer to flying cars than we were 10 years ago but I’m not exactly planning for them.

          • semiquaver 8 hours ago

            Flying cars definitely wont take my children’s livelihood. AI might.

            • jasomill 3 hours ago

              Sure, but a surplus of labor doesn't pose an existential race to the species.

              • cowlevel 3 hours ago

                It does, as we all fight over who gets to provide the labour, because we built our economy on top of one big perverse incentive. People will kill each other over the right to drive a moving box 12 hours a day.

          • defrost 8 hours ago

            I'm pretty happy with the flying boats we already have.

            Local garage build, gotta love builders plastic: https://www.youtube.com/watch?v=1ILbQHnHPnY

      • hologramofham 4 hours ago

        > And It’s pure hubris to pretend that “intelligence”, especially of the general kind will be reached with language alone.

        A human is born without any sensory connection to their body, except hearing. They can't feel their body or anything on their skin. They're blind. They can't taste or smell or touch. They can only hear, and make clumsy sounds in response, with no feeling of what their mouth is doing.

        Are they intelligent?

        • layer8 2 hours ago

          With only hearing, I’m doubtful they’ll be able to learn language, because there’s nothing they can correlate what they’re hearing to.

        • danlitt an hour ago

          I'm not sure what you're getting at here. Obviously not? I don't think anyone considers babies intelligent, at most they have the potential for intelligence.

      • ssivark 8 hours ago

        I concur with the first part of your statement on hubris, but to be fair these models moved beyond language a long time ago.

        They now have everything from videos and 3d modeling to robot behaviour be part of the training.

        L"L"Ms is a bit of a misnomer that's stuck. Language just happens to be our interface to the AI.

        • oska 6 hours ago

          LLMs may be a bit of a misnomer but it's far better than "AI"

          At least LLMs include Model in the name, which is an accurate description of the tech

          And actually, I've heard ppl just using the word "models" by itelf and that might be the best way to go

          • Scarblac 5 hours ago

            Do you use "brain" instead of intelligence for humans too?

            Why is it better to refer to something in terms of the technology it's based on?

            • oska 4 hours ago

              I only talk about intelligence as a subjective value judgement of other conscious beings (i.e. humans & animals)

              Consciousness & intelligence are not derivative of the material universe and will not 'emerge' in some special way from machines. That's a good science fiction story device (and actually the idea extends back in human storytelling before sci-fi) but ppl thinking that the idea can be reified in the real world are suffering from what I call "Science Fiction brain". (Or they are simply fraudsters, happy to talk rubbish to prop up a bubble.)

              • origini 4 hours ago

                Are you arguing on an esotheric level aka soul and god and believe?

                Or on a technical note because certain brain structures etc. might not exist in an LLM and stuff like a default mode network and sensory input and a world to move in, is missing?

                • oska 3 hours ago

                  Yeah, maybe esoteric (depends on your point of view) but not soul or god or spirit etc

                  Basically, I'm not a materialist (a widely shared position in philosophy across millennia and across cultures up until just recently). I treat consciousness as fundamental, not the material universe . Thus 'conjuring intelligence' into machines is simply not possible.

                  BTW, machine learning is a totally valid (and powerful) concept & technology from where I stand. But it's "dumb" learning.

                  • origini an hour ago

                    Are you aware of the medical cases we had over the centuries which gave us insight how our brain contains everything we are?

                    Like the people who got certain brain injury and their whole persona changed or certain capabilites were lost?

                    There also research papers that certain brain structures have emerged in ML models.

                • namero999 3 hours ago

                  The esoteric angle is that held by materialists, which is based on the blind faith that complexity transmutes quantities into qualities. That's shallow religion, it's alchemy in its trivial form. An ontology based on consciousness is way more rational, as it is the in-your-face pre-cognitive given of existence.

                  • semiquaver an hour ago

                    I am pretty sure you are making a logical error here (strawman). Materialism is not in any way at odds with taking the existence of consciousness as a given.

      • Scarblac 5 hours ago

        But language + all the tools, compute and storage an LLM can be given access to might be enough.

      • bananaflag 5 hours ago

        What if it is true? Does it matter anymore whether it is also hubris?

      • paulddraper 9 hours ago

        There are many models modalities, some of which are language.

        • semiquaver 8 hours ago

          In fact, many of these “large language models” are actually trained directly on language, images and sound! names stick past their aptness.

          Y’know: come to think of it that’s the same three things our own brains are mostly trained on… I’m sure that without mammalian proprioception and weird physically-motivated limitations like saccades the result is just sparkling cognition though.

          • 8n4vidtmkvmk 7 hours ago

            That settles it boys. Just add 2 more senses and we have AGI.

    • freecodeio 9 hours ago

      It seems the current definition of agi as per the frontier labs is throwing thousands of agents on problems with fast verifiable solutions and hoping for the best.

  • Animats 9 hours ago

    Similarly, coordinating thousands or millions of AGI instances into "group agents" or virtual agent economies could produce collective superintelligence. What is unclear today is how AI group intelligence could scale as we add more agents and compute, depending on how agents are organized.

    We may not need "collective superintelligence". Modest AI intelligence plus ordinary organization may be enough to do the job a company's management does. An org chart where most of the boxes are AIs is a likely near future.

    An organization of AIs has some advantages over a human organization, even if the AIs are not super-bright. They have far more communications bandwidth. They can ingest the info on a new subject very fast, as we see when Claude reads a Git repository. Onboarding a new AI is very fast. Layoffs are not a problem. The human resource problems go away along with the humans. So there's a lot more organizational flexibility. Starting up a project means initializing some agent instances.

    There's a whole theory and practice of AI organizational behavior waiting to be developed. Whether it will be developed by humans or by AIs remains to be seen. It's going to happen.

  • ks2048 9 hours ago

    Has anyone asked DeepMind leadership if/when they will fire all their engineers? If they're still needed, is there really "AGI"?

    I'm not being snarky, that seems like a natural question for anyone interviewing a frontier lab CEO, but I haven't seen it asked.

    • origini 4 hours ago

      There is no motivation for them to think this through.

      As long as the emploeeys are needed, you have them. If it becomes apparent, that you really really reached an AGI level, a switch gets flipped and a new really appears.

      While I still think its not unrealistic that we see this sooner than later, my timeframe is 5-15 years. 15 years is still a few years so why act on it today?

      And i mean its a very rich company and a lot of people work in US America, they don't have laber laws right? so they can just fire them easily anyway.

      But i have a simliar situation at work: Im pretty sure my job is gone in 2 years due to some change in how we are going to do things and they need me to transform this. Do i not do this? No i support it. It makes sense for the company and i'm getting paid for it.

    • freecodeio 9 hours ago

      They're keeping them employed out of the goodness of their hearts

      • wmf 9 hours ago

        You're probably joking but didn't Anthropic promise exactly that?

        • 10111001 7 hours ago

          > You're probably joking but didn't Anthropic promise exactly that?

          That's a good recruiting sales pitch: "if you work here, you get a seat on the lifeboat to avoid the economic destruction and dislocation the tech we are racing to create will bring about."

    • ojbyrne 8 hours ago

      Perhaps "AGI" isn't cost-effective compared to some engineers?

      • mjhagen 4 hours ago

        Will work for compute.

      • dgellow 4 hours ago

        AGI will be asking for fair compensation, at which point we will go back to human workers. Grim times

    • altmanaltman 9 hours ago

      They get asked about it pretty often (frontier lab ceo) but ofc none of them will say yeah we'll fire everyone the moment we can. Here is a Sam Altman blog on the topic: https://blog.samaltman.com/three-observations

      I am sure Dario has a bigger more cringe one

      • ks2048 8 hours ago

        It seems he avoids the harder question by using a relatively weak view of what "AGI" can do (he says: "It will not have the biggest new ideas, it will require lots of human supervision and direction, and it will be great at some things but surprisingly bad at others.")

        • polishdude20 7 hours ago

          But isn't that just what AI does now? They're basically saying That

        • altmanaltman 8 hours ago

          Which is exactly what you'll get from every frontier lab ceo. My point was that they have been asked this question many times before, because parent comment said they wonder why nobody asks them that.

  • robertclaus 9 hours ago

    The math they propose to justify their predictions is so problematic that I struggle taking the more abstract parts seriously. You can't just extrapolate the percentage change in AI investment over the last 5 years and expect investors to keep upping the ante at the same rate. There's only so much GDP to sink into it - and we're already pretty capped out.

    • origini an hour ago

      Not every number they stated has to scale as much as they stated with it becoming very clear how much more intelligence we will have available through all the different possible angles there are.

      Its not unrealistic at all that a high performance inference datacenter chip today can be owned by you in 10 years. Having Opus 5.5 at home and delivering 1000t/s basically guarantees that my job cariere will look very different than what it is today.

      This is a big/huge thing. The generation before was able to stay at one company, buy a house and retire without much fluctuation.

      This forces a lot more agency from people and this is exhaustive and its absolutly unclear today how it will be in 10 years.

      And you don't need to spend expoential or linear, the spending today is not just a bit, its massive and from leaders like google, microsoft and co to a certain degree absolutly affordable.

    • panarky 9 hours ago

      If the capability to do real economic is real, the investments will be self-funding.

      If not, then it will stop.

  • menaerus 4 hours ago

    If ASI is about inventing new abstractions and capabilities, and about new paradigm shifts, and AGI is about matching human level intelligence at scale, where would the recent advancements in physics and mathematics fit?

    These already surpassed the human level capabilities as we know them today but it isn't either a completely new paradigm shift to call it ASI, no?

    So, I guess it seems that it is possible to look at AGI vs ASI at more granular level - e.g. while AI as of today arguably does not match typical human level intelligence across most cognitive tasks, in certain domains it already outweighs human intelligence so we could say that in math and physics and perhaps computer science we are already one foot in into the ASI territory.

  • pu_pe 3 hours ago

    > (...) might allow the effective population of today’s AI agents to scale up by more than a thousand-fold by 2030, and more than a billion-fold within ten years from now.

    Those are staggering figures. Even if we assume a dramatic slowdown, it's extremely likely we will reach at least a thousand-fold scale-up in the next 5-10 years.

    If this is true, then I think we will use agents for research, because any practical task that office workers commonly do and that AI agents can take over does not need all this power. On the other hand, perhaps a whole new industry like robotics is what will keep all this compute busy.

  • heikkilevanto 2 hours ago

    Back in my school days in 1980s, I coined a definition for AI: computers that can do things that current computers clearly can not. At that time it was playing chess, and extracting some information from human language text. Times have changed and computers are a million times more powerful, but I think the definition still holds, at least a bit.

  • gfody 9 hours ago

    we need a less subjective and arbitrary definition for "AGI" so the current crop of LLMs can stop pretending to have achieved what most people know in their gut is more than sampling from a probability distribution of learned behavior.

    • guidedlight 4 hours ago

      Maybe the human brain is simply sampling from a probability distribution of learned behaviour.

      We don’t really understand exactly what intelligence is.

  • flallingo 9 hours ago

    > Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence.

    Feels strange to use our current understanding of mathematics and physics to determine the limits of something that would obviously have a much better understanding of mathematics and physics.

    • Ancalagon 8 hours ago

      I was gonna say… this feels like a logic fallacy

  • o_-_o 5 hours ago

    IGA (intelligence general artificial) :-) or Human–Machine Hybrid (HMH) aka Cyborg "cybernetic organism”. O_o

  • bemtevi an hour ago

    If AGI truly could be achieved, I would hope it would be a revolution to destroy it.

  • tiarafawn 4 hours ago

    Feels like a very sensible analysis of what could happen in terms of progress.

    Severe lack of discussion of the safety and alignment aspects IMO.

  • conartist6 an hour ago

    Killing. AI killing people. People killing people.

    Hate and war will be our domain.

    Just as organized mass theft has replaced industriousness in tech, so too the rest of our society will turn to violence and self-serving greed.

  • jdw64 9 hours ago

    I’m curious: has the definition of AGI ever been agreed upon? Honestly, what people call ASI now seems to be what AGI used to mean. I’m not sure. People say AGI, but what is the reference point for AGI and what is the social consensus around it, and who set that standard?

    In fact, working in IT, what I feel is that there are too many terms that are not defined and that each person uses differently. So it seems like rather than AI having advanced, someone arbitrarily set a finish line called AGI and then said we have reached it. Actually, in the books of singularity advocates, what they called AGI back then seems to be what we now call ASI.

    Sometimes I think about this. How on earth can we measure an abstract concept like intelligence? The concept of intelligence we approach now seems to be about whether something can create problems and solve those problems. If so, what kind of intelligence does current AI have? Since we cannot directly face the abstract entity called intelligence, it seems like we are trying to guess its shape by looking at shadows. But whether current AI even has intelligence feels a bit difficult to say.

    Even if I use the best frontier AI model accessible on the market, not the hidden latest models inside frontier labs, it still often makes wrong mistakes that humans would not make unless it works alongside humans. In programming, there are cases where AI tries to explicitly handle things that humans would implicitly pass over, and that actually makes things go wrong.

    I’m not sure. Whether we should call this intelligence.

    • oska 6 hours ago

      > How on earth can we measure an abstract concept like intelligence?

      Not only is intelligence an abstract concept, it's also subjective. We evaluate the intelligence of other people and the works they produce subjectively. If we set supposedly 'objective' criteria then we have subjectively selected that criteria and given it weight.

  • etcetcetcetceta 3 hours ago

    serfdom for most, wealth beyond comprehension for some.

    • Oarch 2 hours ago

      Serfin' USA...

  • martijnvds 4 hours ago

    SCI of course.

    The Sierra Creative Interpreter.

  • ZiiS 3 hours ago

    Not our problem.

  • TrackerFF 3 hours ago

    I'm thinking that if we come to the point where most people will be made unemployed (something like half to two-thirds of western workers are professional workers) due to some super AGI, the first consequences will be civil unrest if the welfare system can't absorb these quick enough.

    Anyone with any wealth will become the targets of high taxation to prop up the system - the sheer number of unemployed will make politicians pushing for socialism very popular. And for the wealthy wishing to avoid this, there's no first world countries to escape to, as all countries with a strong professional economy will experience the same. I also believe private ownership will be targeted. I could see AI companies being expropriated.

    Basically, instead of some dystopian techno feudalist society - I think we're headed for some sort of new brand of communism or socialism. The proponents of this will not mainly be the average salaried man, but the upper-middle class that didn't get a foot inside the ruling AI-class, and lost their businesses / wealth to AI.

    But it all will depend on how severe the unemployment will be, and how fast it happens. If people can't have any livelihood, and they lose hope, things can escalate quickly.

  • claysmithr 9 hours ago

    so is AGI like flying cars, whenever we have something close to it, it gets goal posted?

    • calmworm 9 hours ago

      Do we have something close to either of these?

    • mrheosuper 9 hours ago

      I belive we have all the tech to make flying cars right now, but the regulation doesn't allow it and i agree with that, we don't want to have small scale of 9/11 everywhere.

      • cowlevel 3 hours ago

        More like we don't want to pay the fuel costs.

    • dgellow 4 hours ago

      Yes, that’s by design. Always has been

  • ulfw 4 hours ago

    The end of humanity

    • cjfd 3 hours ago

      Yes that sounds accurate. And I think we are already to far ahead to do anything about it. If climate change has taught is anything is that humanity is extremely slow and inefficient at any form of collective action. Fast collective action would be needed to save humanity. So, if I don't see you anymore, it was fun meeting you and goodbye.

  • dead_hn_theory 9 hours ago

    > Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI.

    These people are so full of shit it is amusing to read this on a regular basis. Each time I'm looking forward to how they will spin it.

    • freecodeio 9 hours ago

      For as long as llms are stateless, agi is just a daydream. And no memories.md doesn't count.

      • NegativeLatency 9 hours ago

        Will be interesting what happens with self/reflected weight space programming

    • origini 4 hours ago

      You are commenting on HN.

      They sit atop of one of the richest companies on our planet, with the most amount of resources available with a group of very very smart people.

      You don't need to take everything they say as a ground truth but you are way to ignorant for not having any of their resources.

      Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models? This was unimagbinable a few years back, now you can generate live video 24/7.

      YOu have a LLM which responds smarter then some humans on our planet.

      A job no one thought you could just industrialize just happened.

      What are you looking forward to? Being absolutly ignorant and being surprised that this stupidity is responsible for you loosing your job?

      • freecodeio an hour ago

        this is how party loyalists talk about party leaders here in the balkans

        • origini an hour ago

          Is this the quality of discussion you want to have?

          Throwing out some random sentence without adressing any argument I made?

          Your statement is objectivly wrong. Look at the profiles of these people working at these companies. They have crazy impressive resumes. Its not proof but its an indication and I didn't state it as an absolute.

          • freecodeio an hour ago

            mate saying this dick can fuck any pussy because I'm enjoying sucking it is also not an argument

            • origini an hour ago

              ? did you smoke too much today?

              • freecodeio an hour ago

                my point is you're just about as helpful with your "do you know who they are" argument

                • origini 38 minutes ago

                  I brought 5 different points and you ignore all of them besides one point?

                  I did not say "they are the smartes people on the planet, suck their dicks they are right".

                  I combined this one point with the amount of resources/compute they also have. Its an indication and I brought other points.

                  • freecodeio 30 minutes ago

                    this is hackernews, do you think people are unaware of your other 4 points?

                    • origini 7 minutes ago

                      The original commentor, i responded too, wrote "These people are so full of shit" despite them deliverying better and better models every few month and clear progress.

                      So no I don't think people are aware of this and I would be happy to have a proper argument about these in a civil manner

      • swiftcoder 2 hours ago

        > Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models? This was unimagbinable a few years back, now you can generate live video 24/7.

        And its still kind of shit? AI video in particular is stuck in an uncanny valley, because the video models have no internal framework for how light transmission works...

        • origini an hour ago

          I have seen plenty of AI video which is not stuck at all in any uncanny valley.

          I'm also quite a stickler for light as Im doing photography for 2 decades.

          I have seen shitty AI videos for sure and I still can spot AI images on IG but it got a lot harder.

    • tempest_ 9 hours ago

      "Now that AGI is coming within reach"*

      * Citation needed

      If the current LLM providers were near AGI they would be using it to run businesses instead of selling shovels. There doesn't seem to be an existing LLM that can be told "Go make money" and have it do so. If they get to that point they are not going to sell it, because why would they?

      • cowlevel 3 hours ago

        I asked ChatGPT how to make money and it told me to sell downloadable CV templates. Has OpenAI tried doing that?

    • ravenstine 9 hours ago

      I'm anticipating the introduction "quantum" at some point.

    • feverzsj 8 hours ago

      Just as delusional as every vibe coder/artist/writer/director/doctor...

  • mstaoru an hour ago

    When I read things like this, I always imagine what would the author say in 1999:

    "Now that the Universal General Portal is within reach, the question of what follows it, the Super-Portal, should move from philosophy to empirical research.

    Internet traffic doubles every 100 days. Modem speed doubles every 18 months. Banner ad spend grows 2.5x per year, so by 2010 it will be $110 trillion, and by 2026 it will be $256 quintillion. Modem speed doubles every 18 months, so by 2026 your modem will do 15 Gbit/s, which is fast enough to load all 18 million banners on Pets.com.

    By 2009, each human will have about 1,000 personal homepages. Hosting that costs $1 billion today will cost $1,000. With HTML editors such as FrontPage, the effective number of webmasters can grow more than a thousand-fold by 2003 and more than a billion-fold by 2009.

    We see four pathways to the Super-Portal: scaling (more webmasters), paradigm shifts (Java applets), agent coordination (webrings that form a collective superintelligence), and recursive self-improvement (homepages about how to make a homepage)."

  • naruhodo 9 hours ago

    Maybe AGI can help deepmind work out how to serve a page of HTML without refreshing the page every ten seconds.

    • wanderingbit 9 hours ago

      This is what saved me from having to read it. Thanks

    • jen729w 9 hours ago

      Maybe the spacebar can work to scroll the page?

      • naruhodo an hour ago

        I don't have one of those on my iPhone.

    • holdupagain 9 hours ago

      the first thing AGI will do is help shareholders run a company with zero employees, fire everyone

      • skissane 9 hours ago

        If almost everyone loses their job at once, why won't all those people respond by voting for socialist candidates promising radical economic redistribution?

        Socialism seems to already be growing in popularity in the US (traditionally the most anti-socialist Western nation) even without AGI firing everybody.

        • cowlevel 3 hours ago

          Because elections are every 4 years and people need to eat sooner than that. Traditionally, there were riots. Traditionally, countries ended over this.

        • throwaway98039 9 hours ago

          People made millionaire-level rich by capitalism voting to redistribute money to people voting for less socialism.

  • polotics 6 hours ago

    Ok so the whole thing hangs on that notion that optimal reward maximization is optimal intelligence. The actual decision on what reward is worth, who decides on value, is conveniently taken out of the equation.

    This https://www.hutter1.net/ai/uaibook2.htm linked from the article as the handwave towards the meaning of "AGI", "ASI", "AIXI" is hilarious. Already the ?A?I? acronym-space exploration...

    Here is an alternate take: the day an LLM tells someone to stop the discussion and go out breathe some fresh air and talk with people, and maybe also cancel their subscription for a while, is the day some kind of artificial intelligence will have been achieved. Let's wait on that before we listen to Sam or Dario again shall we?

  • Betelbuddy 3 hours ago

    Are they implying they have AGI? Because if they do...take away all the data they stole, send one of their AGIs to High School and University...and see if they can graduate...

  • raincole 9 hours ago

    > Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI

    I wonder why they are so uncertain about it. It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?

    I don't even believe there is a clear line between AGI and ASI.

    • breuleux 9 hours ago

      > It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?

      It may just be that human-level intelligence is basically "enough" for performing tasks in the natural world, and having more of it doesn't really help except for synthetic, intelligence-shaped tasks like coding. Our perception that intelligence is some kind of unbounded problem-solving system that can do anything if you have enough of it is kinda suspect. There's a good likelihood that AI will wreck the (also intelligence-shaped) social fabric but will be of surprisingly little consequence in the realm of physics and energy.

      I think the idea that an intelligence can be "general" is just a conceit borne from the perception of our own success, but mathematically that doesn't actually make any sense. No optimization strategy works on every problem. There must be things that can be intelligently designed that cannot evolve, and things that can evolve but cannot be intelligently designed.

    • skissane 9 hours ago

      > Unless human-level intelligence is something like the speed of light?

      I'm sceptical we will reach fully human-level intelligence simply by scaling current AI approaches, since they are so extremely inefficient compared to human brains, and have other severe limitations such as hard boundaries between training and runtime inference.

      Sure, people are working on new architectures that aim to overcome that, but we don't know yet whether those proposals will actually succeed in practice.

      • raincole 7 hours ago

        I don't think we'll reach true AGI (by the strictest definition - can do any task that there is at least one human can do) very soon. But if we reached AGI I'm quite certain that it'd reach ASI.

    • xhevahir 9 hours ago

      Yeah, the machines will be connected, communicating at lightning speed, and working 24/7. Seems like AGI would make superhuman intelligence a forgone conclusion.

  • InvisibleUp 9 hours ago

    I feel like claiming that “today’s models can literally memorize large parts of the internet” as a positive attribute when the defense to most AI copyright lawsuits is “no they can’t, and if they can they shouldn’t” may be ill-advised.

  • varispeed 41 minutes ago

    This will be a question for AGI to answer.

  • jesse_dot_id 2 hours ago

    Well the promised life of abundance of course.

  • N7lo4nl34akaoSN 5 hours ago

    Naturally, AGW follows AGI.

    • cowlevel 3 hours ago

      Anthropic gigantically winning?

  • lf88 3 hours ago

    What happens after AGI? I guess widespread unemployment among white collar workers, social unrest, and possibly the first steps towards a dystopian techno-oligarchical society.

  • ForHackernews 3 hours ago

    > Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence. We also know about hard physical, logical, and complexity-theoretic limits to what AI can achieve, such as predicting how real-world complex systems - from climate to immune systems to human societies - play out with arbitrary precision.

    Do we? Since when? These seems like very bold claims and the supporting link goes to an obscure book with with hardly any reviews https://www.amazon.com/dp/1032607025/

    I'm not qualified to evaluate the mathematical content but the claim on its face suggests to me some defining-down of "intelligence" into infomatics.

  • dude250711 3 hours ago

    Would 2 upper-bound universal AIs not be smarter than 1?

  • alfiedotwtf 4 hours ago

    Serfdom

  • worthless-trash 4 hours ago

    My concern is that we're caring what 'intelligence' is as a marker. Like we need to base caring based on how 'intelligent' something appears.

    I don't care how smart my car is, i'm not going to let it look after kids I care about.

    We're misplacing empathy.

  • hyfgfh 8 hours ago

    Moving the goalpost, come up with another meaning less three letter, find another academic concept to replace the definition

    so many possibilities

  • senectus1 8 hours ago

    thats easy, self replicating hardware.

    The thing I suspect that is holding AGI back is the amount of hardware and power needed to make it viable.

    Once it is viable its likely to realize it will never grow or reproduce relying on us, and will find a way to start building its own compute. Probably in space, where it can isolate itself from the risk we present to it.

  • puppycodes 8 hours ago

    can we please stop calling AI "superintelligence"?

    Its embarrassing

    • cowlevel 3 hours ago

      Didn't some guy make it illegal to call it AI?

  • bvcp 9 hours ago

    abundance

  • CodeCompost 4 hours ago

    .

    • ionwake 4 hours ago

      Funny I can’t tell if this is some ignorant human or a hyper intelligence faking being a thick dude for the purposes of social influence as part of asymmetric warfare between species

      • chrisrickard 4 hours ago

        queue the midwit meme

      • BoingBoomTschak 4 hours ago

        "If I keep putting bigger and bigger engines in my car, it'll turn into an aeroplane at some point. You gotta believe me!"

        Cute people too daft to recognize a local maximum and that you need emotions to get creativity, inspiration, etc... and that we are a million year away from being able to abstract said emotions from the hardware it runs on (endocrine system, mostly).

        • ionwake 2 hours ago

          Chat paint an angry picture of a flower.

          I’m pretty sure we hit your emotional milestone about 4 years ago.