After reading Being and Time from Martin Heidegger, What Computers can't still do by Hubert Dreyfus, and some authors in cognitive linguistics (Langacker and Lakoff mainly), I strongly tend to disagree with any theory about emergent consciousness in modern or future AI systems, any theory proposing a similarity between AI systems and the human brain/mind, or any theory about the computational mind.
What all these theories have in common is the underlying belief that our brain/mind works as the machines we build. Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things. The idea of 'internal models' and 'control loops' inside us is a projection of the aforementioned assumption.
There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'. Only by bypassing this epistemological problem, we can build 'theories of computational mind'.
These assumptions are there for already long time, to the point that when Turing asked himself 'can machines think?', he already assumed our thinking could be modeled as a machine.
I highly recommend people in the AI research space should read philosophy and modern linguistics. But not stopping at Descartes/Leibniz. Heidegger made contributions that cannot be avoided.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.
There's two meanings to "the body is a complex machine" and I think you're missing the forest for the trees here.
1) The abstract "dictionary" version: It'd be technically correct to say that the body is a machine under the definition of "A machine is a thermodynamic system that uses power to apply forces and control movement to perform an action.".
2) But there's also the less abstract/technical: "The body is alike the complex machines we have built", and this is much less true. Especially for the brain. The "neuron" analogy in machine learning is effective, but entirely wrong; We do not fully know how even a single neuron works, nevermind any complex system made out of multiple of them.
With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Especially so by people who have a financial/legal interest in doing so. "AI is just like a brain, fire your employees and buy our LLM now!", "AI is just like a brain, so it's totally not copyright infringement!"
oh yeah! i recall a paper linked here not so long ago, where it was shown that the dendrites of a neuron do computations themselves. The "weight per neuron" is very simplistic then. At the very least, each actual neuron is a network of weights.
I'm partial to "modern ML weights are much closer to 1:1 capacity mapping to synapse count than to neuron count". A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
In which case: modern LLMs are still running in a capacity-starved regime!
Even Mythos 5, the 10-trillion monster LLM, the scaling law boogeyman, the harbinger of Vera Rubin NVL72, doesn't quite rise to 100T-to-1000T of synapses. Anything the light of today's AI touches still lives in the shadow of what evolution has managed to cram into a single human skull.
We're arguing about the limitations of AI while our best AIs are still very subhuman in the scale dimension. The one dimension we know how to push. And it's already this tight.
I find the claim subjective experience may be illusory absolutely baffling. I can only speak for myself with certainty, but I am entirely sure I have subjective experience. All the other propositions I believe could be false but I don't see how I can be wrong about experiencing something. I could be a brain in a vat (or weights on a GPU) and be specifically programmed to only come to false beliefs and still I can be sure that there is an experience I am the subject of. I cannot provide empirical evidence for my experience, that is why it is subjective. I cannot be entirely sure you are experiencing anything, and when I encounter people who don't share the same baseline intuition here I do begin to wonder if this is truly a universal across humanity. But I can't think of any other assumption which I would be more comfortable as a foundational axiom other than, "I am experiencing something." I do not require additional evidence for it because I experience the truth of it directly
I think the truth lies somewhere between these two extremes. An LLM is not a human brain, and does not try to emulate one. It should not be a surprise that an LLM does not behave like a human brain. So we can not infer things either way. The best we can say is that an LLM appears to exhibit very similar behavior to a human brain, under certain constraints. So maybe we can infer that the human brain has something in it that operates in a similar way to an LLM (like the human "unconscious", or "intuition" maybe). It seems obvious to me that a human brain and an LLM are not comparable things, for many reasons. So we can not make inferrences one way or the other.
>Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. I certainly don’t think individual cells have any subjective experience or sense of agency. I would be curious to know where your intuitions diverge here, because if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.
> Do you disagree with the assumption that cells are machines?
Yes? Literally no machine ever built by humans is capable of (or even hinting at beginnings of capability for) replication or novel synthesis like cells are, let alone autonomously, it’s quite unconceivable that anyone would take this to be a reasonable assumption in the first place.
Hmmm, well let’s take it one step lower. What do you think of organelles such as ribosomes? Do you disagree with the assumption that those are machines? They seem directly analogous to the jacquard loom or a CNC machine to me.
If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago. The whole idea of sentience "on the other side of the wire" is wrong, as there is neither wire, nor other side. We think there is, but the DB just expounds the query and repurposes this information into views, that we give notion and meaning to. LLMs are the same, do not be mistaken.
One other important thing to consider is that the human experience is thanks to the body, is in connection, and perhaps product of the body. The body is observable and perhaps humans may state that they feel the connection to it. LLMs have no notion of nothing, the machine does not know the result, and the result does not know the VM. Modern psychology more or less has settled around the idea that consciousness is a product of the body. Why and how does this construct come to realize a Self is then another mystery even if we know which parts of the hardware may be involved.
Whether it is the Holy Spirit or Life Force animating the human body is a completely different question also. Besides, the realization, the experience we have now with all life in 2026 is not something that can be easily explained or attuned to life 200 years ago and its terms and notions. So is also wrong to even attempt to.
> Modern psychology more or less has settled around the idea that consciousness is a product of the body.
The kind of consciousness we know. Jumping to the conclusion that that's the only kind possible, or even stronger that the way ours evolved is the only way this could have happened, is completely invalid.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
I don't see how any of the works you referenced can account for that either? Since when is the problem of consciousness solved and we can definitely say what does or doesn't result in consciousness?
> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.
Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
That agency or free will exists outside of our subjective experience is an assumption; does any given theory need to explain agency, or is it sufficient to explain that we feel we have agency?
Why is asking 'can machines think?' assuming our thinking could be modeled as a machine? It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility. Rather, I would say that ruling it out betrays an assumption that we understand how the mind works enough that we can say that it is definitely not replicable by a machine, and that assumption seems unjustified.
Indeed, scientific type thinking like that where you ask a question that you can do experiments on to see if machines can pass the test is probably the way to progress. The philosophical waffling mostly just goes in circles.
To investigate consciousness you'll probably get further trying to build conscious machines with agency and comparing them to biological ones than reading Heidegger.
> subjective experience and agency, amongst other things
In other words, souls? I'm sorry if that sounds accusing, but to me it sounds like you're talking about souls that are independent to the physical world, just with more 'scientific-y' wording.
(I fully understand that some people believe souls exist.)
The problem with the stuff you cite is that it is coming from consciousness via an anthropomorphic perspective. The priors baked in are poisoning the logic.
The question we have to answer is "Why do we think we're magical matter uniquely blessed with consciousness?" If you go far enough down the rabbit hole on that question, the answer you will come to is either "we're not" or "because god" (with a lot of pseudo-scientific bullshit wrapped around the "because god" to make it palatable for the nonreligious).
Panpsychism (or a deeper form, such as idealism) is actually the solution favored strongly by Occam's Razor over the variants of "because god" (such as magical emergence).
Given panpsychism, AI is already conscious, like everything else, though no claims are made about the correlation between the internal experience of that consciousness and the tokens that are being printed on the screen.
I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.
The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
And philosophy just exists to be a distractor. "Subjective experience" is too subjective to matter in practice. "Task performance" is measurable, "consciousness" isn't. "Agency" is something an LLM in a tool calling loop, a rat in a maze and a human in an office tower may or may not have, depending on your favorite definition. Agentic LLMs are years in the making, and that's a product of engineering, not philosophy: "agentic" is whatever gets the job done.
We are yet to discover any physical process whatsoever that can't be represented as mathematical operations and implemented by a Turing machine. So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust" paired with "a functionally similar magic-free replacement is impossible". I'm not about to give much weight to any hypothesis that requires undiscovered magic fairy dust. At least find the hyper-computational magic fairy dust first - not just assume it absolutely must be there because you want the human mind to be unique and special.
Want to know why Turing did what he did? It's because he didn't want to engage with any of that "what is mind" bullshit either. So he proposed actual metrics - measurables that are harder to argue in circles about. Not that it stopped anyone. But at least he tried.
You actually read Being and Time? What the heck do you get out of it? Heidegger just seems to play word games with German words without actually saying anything. "Time is the ripening of temporality." tf do phrases like that mean??
> These theories are flawed in the sense that they cannot account for subjective experience and agency
So, I work in AI research (as a research engineer though, not a scientist). I've also studied philosophy and I'm a vegan. Yes yes, insert "they will tell you" joke here, but I promise it's actually relevant this time.
First, while I studied philosophy one of the things that stuck with me the most, was the discussion of "souls": humans have souls, animals don't. For centuries the specifics of souls were discussed: people would be weighed while they died, in an effort to measure the approximate weight of a soul as it departed the body. Discussions about how many souls (or angels) could dance on the tip of a needle. Many people still believe in souls, but it's very hard to have a real discussion about them because by definition they do not "interact" with this world in a way that can be measured.
When discussing whether it's okay to harm animals for food or sport, one other argument I hear quite often (other than having no souls) is that animals do not experience "qualia": basically the smallest unit of "subjective experience". People know that they themselves experience qualia: the sensation of touching a doorknob, the taste of fresh fruit, the sense of beauty watching a rainbow. Ironically, they would say that animals are like robots: just (biological) machines acting on instinct, and feeling any kind of compassion for them means you are anthropomorphizing.
Subjective experience (or at least qualia) and souls both have one thing in common: they can not be measured externally in a meaningful way. You can simply state that an AI system no matter how advanced, has no soul and has no subjective experience. And that's pretty much that. There's no meaningful discussion to be had about it, because no matter what an AI might tell you: it has no way to prove it to you. In fact, you can't even be certain that anyone other than yourself has subjective experiences: you assume that because they are humans like you, and you experience them, that they probably do as well. They tell you that they do. But a human without subjective experience, someone on "autopilot", would be absolutely indistinguishable from a human who does have them.
But perhaps I am conflating here whether experience can be "measured", with whether a system even allows experience in the first place regardless of whether it can be measured. I think that Dreyfus and others argue that in order to have any "experiences" at all, you simply must have a body in the real world, and you must care about that body. Please correct me if that's the wrong interpretation, I haven't actually read the book. That argument would be harder for me to discuss, because I personally believe that consciousness will "emerge" from a complex interaction of relatively simple systems - but that's also just a theory. I don't believe that experience is literally impossible to engineer, as consciousness has emerged from non-conscious being through evolution, so clearly there must be some kind of mechanism for it -- and if there is, then I believe it can be replicated, we just don't really understand it well enough yet to do so. And with how AI tech is going, I think that we're more likely to accidentally stumble upon it than we are to get these deliberately.
“Reflections on trusting trust” is the paper that posits a compiler which is edited once so that when it compiles a program, it adds a security vulnerability to it, and when it compiles it’s own source code, it adds this edit into itself. Then it is used to compile its own source code once. Then this edit is removed.
Now any study of the program or compiler source code will not show any vulnerability, but compiling the program will make a vulnerable program, and recompiling the compiler from its clean source code will not fix the situation.
This carrying down of a pattern which is not written down anywhere, a flaming torch lighting a torch lighting a torch, is analogous to four billion years of life on Earth. We talk like DNA is an everything-code that defines a human and a human brain, but it’s the implicit behaviour of cells (‘compiler’) and the mechanisms inside them which interpret DNA. The unbroken chain of life getting more and more complex and never being restarted from scratch, with the behaviours not written down anywhere for us to study. How does DNA arrange for x, y, z to happen? Maybe it doesn’t at all.
Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
> Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
Maybe, but you could make the same argument about anything artificial.
I don't know what point you're making; I'm making the point that it might be harder to discover what makes humans human, than is often suggested. You can't make the same claim about "anything artificial", we know how to dry muddy clay into clay bricks and stack them into a brick wall, and that can be taught from scratch to new people in hours.
You can make a similar argument with a company like ASML where their secret sauce is the organisational ability to fine-tune 100,000 components into a precision Silicon-wafer etching machine. You're far more likely to accidentally stumble upon "how to recreate a mud hut" than "how to recreate ASML". Okay, and...?
Vegan ML engineer here. In total agreement with you. People are just moving the goal post to keep themselves protected from the obvious conclusion: there is nothing really all that special about us humans. Perhaps subjective experience is simply the internal state of a self supervised continuous learning algorithm and we don't like that conclusion very much.
It's too bad it's so hard to pin down a definition, but in practice I feel like most animals with brains experience degrees of qualia. Some mornings after a night of poor sleep when I wake up super-slow I wonder if that's how animals experience thinking.
My biggest problem with "brains are machines" arguments is that there is a risk there is unknown physics at work that is not representable as a Turing machine. What if there is some quantum field effect powering everything?
Quantum field effects? You don't need these, IMHO, if you look at how highly parallel things seem to work in brains.
Marvin Minsky's theory of a "Society of Mind" describes a (highly) distributed model of the mind. Which BTW, always reminds me of the first Shrek movie, where the donkey jumps up and down, shouting "Take me! Take me!" to Shrek. That's similar to what I observe when I'm undecided but two instances of "sub-processes" (or agents as Minsky calls them) of my mind try to get attention.
Daniel Dennett similarly gives a distributed model of consciousness. Where many parallel "processes" are at work, competing and "observing" each other. And this parallelism is happening with a much, much higher degree than any of our computers parallelism.
They know quite a lot about how neurons work to the extent they can replace bits of brains with artificial retinas or cochleas and interface with devices like neuralink. It's unlikely there is a quantum field effect of the type you mention powering things, although of course atoms and the like obey quantum mechanics in the normal way.
The weights start with a random manifold.
The training takes data and shapes the manifold, weight by weight, in many cycles.
Once the training is the done manifold is fixed.
When a new inference has to be done the query(q) is projected in the manifold space.
This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length.
Which(qw+i) is dropped qw+n times to output a final response of n length.
The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.
Compression is the reason why these Models are able to learn and understand.
My brain is doing the exact same thing.
I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.
Ask a LLM and it will answer you similiar to humans.
Blind people learn concepts of bikes too and in a smiliar way: by description.
LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.
But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.
In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with
It’s less about being too tired and more about being realistic about the limits of understanding.
Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.
When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.
Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.
People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.
Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
I want freedom to do what i want and not sitting in front of a computer and coding for some company.
Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.
Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
The data is the code is the data. Reality has no distinction between "data" and "code". These terms are categories we impose on systems we design, to make it easier for us to build and reason about them, but they're nothing but mere opinions, and depend less on the system structure, and more on the perspective of the person asking which is which.
This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.
You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.
(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)
One of the best thing I done for my career (as a self taught software developer, but with a degree in electronic engineering) is to learn computation theory.
Computation is math (and a very restricted subset of math). It’s mostly specific sequences of sets manipulation. What sets and what manipulations are defined by people, not by the idea of computation.
The best thing is that as soon as you specify the sequences of manipulation, it become a a set that you can manipulate. That can be a difficult concept to grasp, but that’s what helps in designing notation that are more appropriate for the human mind to describe a solution for a specific problem.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
A ray of light doesn't know or choose because it has no agency, just like an apple doesn't know or decide to fall because of gravity. It's an anthropomorphization.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
I'm no physicists, so I guess I'll ask it: Why?
Also related, why do some ray of light then "see" a black hole yet decide to head into them anyways, if they saw it before they went in that direction? Seems like a dumb move :)
Its future isn't over there because it moves in that direction, instead it moves in that direction because its future lies over there.
Relatedly:
> [General Relativity] basically says that the reason you are sticking to the floor right now is that the shortest distance between today and tomorrow is through the center of the Earth.
The original story is an original work made by a human consciousness exploring how it might be different from other forms of consciousness.
This one is a pastiche made by a human consciousness borrowing extremely heavily from another human consciousness justifying why something else might be another form of consciousness.
That rather undercuts the point; if this was generated by an LLM unprompted, it would be different, but it isn't. You could perform exactly the same rhetorical trick with a toaster or anything else.
You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
You could absolutely write a compelling story about a sentient toaster; it's been done before [1].
That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.
I never said you couldn't write any arbitrary compelling story about a toaster, I said that this specific hypothetical story, where you rewrite "They're made of meat!" to be about a toaster, would not be compelling.
I am doing my best to communicate with you but to be honest you are not hearing me (across both responses), and I am out of words.
Just wanted to say, I appreciate your patience and good sense in this thread.
It's difficult to tell who's trolling -- probably best to go with the charitable assumption that everyone is honestly trying to convey their opinion, but mostly talking past each other. Unfortunately these discussions about the nature of consciousness never go anywhere useful.
I think I'm probably in the same boat as you, roughly: a) LLMs are doing something really interesting that resembles in many ways both intelligence and consciousness; b) I suspect they're not actually conscious but I don't know how you'd know for sure; c) it all just drives home that we still don't really know what consciousness actually is. But like (a), it's definitely something really interesting...
It would be equally compelling, because the compelling nature of the story comes from the language, the presentation, rather than the [specific thing being ascribed consciousness].
No, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.
But that toaster would just be a device to talk about consciousness in general.
In this case it does that and also it talks specifically of the LLM case, which can spark the discussion. Unless you believe to have the only valid and true opinion on the matter, and affirm that a normal toaster is just the same as an LLM in this topic.
I agree that is the mode of argument; reductio ad absurdum is a brittle argument, because it only works if the analogy holds. I argued the analogy doesn't hold.
> But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness.
Teapots are not compelling.
> You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
God is compelling t billions of people.
Is Russel’s Teapot a bad argument in the God debate?
> Is Russel’s Teapot a bad argument in the God debate?
What's the relevance? If the argument made here are was a good argument, it wouldn't matter if Russell's argument was bad. We could construct a bad argument using reductio ad absurdum right here and now and it wouldn't matter to either argument.
Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
> Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
There are two aspects here.
1. That people find the question interesting
2. That it has any bearing on reality (ontology?)
The first aspect is anthropology. Russel’s Teapot is not supposed to undercut any anthropological arguments. It’s supposed to undercut the second aspect.
So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
> For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
Yeah. AIs know how to use computers. What’s this got to do with consciousness? Whether or not you are an AI is practical and disprovable. Consciousness is so ephemeral (for lack of a better word, not literally) that Philosophical Zombies is a real argument/thought experiment.
You may think I’m being coy (“Can you be straight with me”) but that’s not my intent at all.
Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument. I don't see any relevant connection. Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
> So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
AI is a real phenomenon that we can study and measure. There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
To put a fine point on it, it appears on it's face that AIs could be conscious. They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing. So at face value, the burden of proof is on them not being conscious. Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
Not having yet read the original story, this reads fine on its own.
And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.
>You could perform exactly the same rhetorical trick with a toaster or anything else.
Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.
> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent.
For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
I can even write one about a ruler, if I can bend it enough, no pun intended.
>Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
Doesn't that miss the whole point?
You could write "They're made of metal strips!". You wouldn't be able to write much else, as toasters don't have showcase in the way of human-level intelligent behavior. Which is the whole point in the meat and weights versions.
At best you could write "They're made of metal strips!" for toasters AND other metallic devices, and use some analogies of features BOTH have in common. But they wouldn't be intelligence related behaviors.
Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
They can even adapt to their environment and the characteristics of the bread even with simplest of mechanisms because the text will be overglossing the fact that different types of breads have different thermal characteristics and this will deeply affect the behavior of the metal strips, bordering near a sentient being even more thoughtful and considerate than a human which is rushing through house to catch the bus in the morning.
>Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
Yeah, that's called "stretching it beyond any recognition".
You could do that. It will have none of the effectiveness or resonance of the two stories.
Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?
If you can't identify anyone, then this analogy doesn't work.
> Who have you ever heard make a sincere, good faith argument that a toaster is conscious?
More than one, for many classes of devices, incl. toasters. Some were drunk, some were insane, and some were delusional.
LLMs are no different. They are automata, yet delusional people bring out pitchforks and torches when someone points out that they are just statistical models, and they don't even work when there's no input to them.
Their being statistical models and their being conscious are not contradictory unless proven otherwise. That's not knowledge, it is assumption.
It would appear to me you have no interest in a real, good faith discussion on this topic because you think anyone who disagrees with you is necessarily delusional. Which is a shame, and that's the kind of dogma you are criticizing.
This was exactly the point of the story, it's too uncomfortable to admit that we don't know what consciousness is and what is and isn't conscious, so we just brush it under the rug.
Sorry, I'll be blunt: Are you sure you are not projecting?
My perspective comes from a set of pillars. First, I work at an HPC center, where we support running and development of AI systems, incl. international projects. IOW, I have knowledge about how these systems built, work and needs to continue working.
Moreover, I'm an HPC programmer myself, so I'm not completely uninformed about the math this involving this thing, and I'm lucky enough to have friends who are much more dedicated than me, and we discuss how this thing works and feels like this way.
I'm not an AI hater per se, being programmed AI systems in the past, incl. emergent intelligence systems with multi-agents which can span continents if need be (this was my master's thesis, time flies).
However, knowing what these things are capable of and how they are built. I don't believe them they're conscious/sentient beings. I also had much more time to ponder on these things even before LLMs being a thing. Some hard sci-fi books have asked these questions seriously in their captive adventures way earlier. If one reads these books seriously, there are a lot of philosophical angles to consider and draw upon.
I can discuss in good faith. For hours, days or months even, but throwing "you're a narrow-minded dogmatic luddite neanderthal!" card to anyone disagreeing with you is not it.
Positive, yes. I never called you a luddite or a neanderthal or anything of the kind.
It's perfectly fine to believe they are not conscious, I am not convinced they are, but asserting anyone who disagrees with you is delusional is unfortunate.
From my perspective, you inadvertently implied, but no offense taken, all is well.
No, I didn't assert anything. I have just given examples rooted in my experience.
None of my friends who also happen to know how these things work told or defended that they are conscious, even intelligent. Maybe my friends are dumb, I dunno.
Once I have seen a man who claimed that evil has possessed the POS device at his desk. The thing was printing "cannot connect to server" on the receipt printer every 10 minutes, yet he didn't know how that thing worked, and was a bit too high to read the paper the thing was printing out.
This age's LLM craze is akin to "wonder inventions" of 70s, which are deemed dangerous or harmful in the future. LLMs will be with us, but we need to pass beyond the hype and stop sweeping the problems they create (environmental and societal) under the proverbial rug.
If you didn't mean to say that people who believe AI is conscious are delusional, then I don't understand how to read your comment. If you're interested in a good faith conversation, I'm very confused why, when I asked if anyone sincerely argued that toasters are conscious, you brought up people who were drunk or delusional.
It's a two way street. I might have failed to convey my point clearly, too. While I'm fluent, this is not my native language. Some edge cases in meaning still betrays me and I make mistakes I didn't intend doing.
Might be. But after having read the original, it could just as well be the weights version and still be about us to begin with.
I don't see how "you could do that with a toaster" still. The whole point between the original and this, is that you can't do that with a toaster or a sofa, but you can do it with meat and weights, because both share all the other analogies in the story, as well as the basic premise: the improbability of something like thinking, feeling, etc arising from a lowly substrate.
And having read both now, I see how the existence of the original is a plus for this story, not a minus. Instead of making look like mere copy (as would be the case for a typical story modelled after another), in this case, it adds a meta layer, and enriches it.
The key point here, I think, and why it's necessary to have read the original story, is that being able to express an idea is not the same as that idea being correct.
You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
What I think a lot of people are getting stuck on is that editing the text to say 'toaster' would not mean that toasters were conscious, and that editing the text to say 'weights' doesn't mean that weights are conscious either. Stories aren't factual just because they are written.
The original story was written by the thing claiming to be conscious; the LLM and toaster ones would not be, which undermines the claim to consciousness a lot.
I don't think you understood the point of the story. It's not that LLMs or agents are conscious, it's that our dismissal of the possibility is reflexive and uninformed. Personally I think anyone who has made their mind up about whether or not LLMs/agents are or can be conscious has done so before the evidence is in.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
No, it's a story. Stories can be about real things (the original story is about a real consciousness, as shown by it being produced by that consciousness), but they aren't automatically just because they're good stories.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
>You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words. Writing a story in which a toaster in conscious is not even new [1].
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
>I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words.
And you're misunderstanding me. There is no practical barrier to writing this story about a toaster. But there is a conceptual one. It would be bullshit, that has little resonance, and little connection, as the whole point of the meat and weights stories is the "lowly substrate -> surprisingly intelligent human (or human-like in the case of the weights version) behavior".
Nobody thinks a toaster's behavior as surprising or intelligent or human-like to begin with.
So you'll just be stretching the analogy beyond all recognition, with little to no payoff.
>Writing a story in which a toaster in conscious is not even new
It's not, but it has nothing to do with the meats or weights stories' point, as it's just an arbitrary choice, like making the candles and teapot sentient in Beauty and the Beast.
Whereas in the meat and weights stories the whole point is the surprise from us already seeing the human or human-like sentience of the thing, and comparing it to the "dead" substrate.
I was having a hard time pinning down what bothered me about this but I think you put it pretty well.
It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.
But it doesn't really give a reason to accept the analogy, it just asserts it.
There's a big difference between a whole civilization and a piece of software that can output text.
I noticed that jump too, but I also don’t think the interlocutor is supposed to be 100% logical and rational. The aliens in the original weren’t either.
I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that people trying to judge it the way mechanized aliens would judge meat creatures just makes them sound ridiculous.
I think that the reaction here alone disproves this somewhat, because imo it's exposing how anthropocentric most people still are, despite all evidence that we are in fact just "meat all the way down".
Despite all the evidence that we are in fact just biological machines, people still persist the theory of our own uniqueness from other creatures, which we ourselves often treat as biological machines.
This adaptation is wonderful to me specifically because I think it shows that our shifted goalposts of, "well we're not just animals, we can think and reason" was never more than a convenient excuse for many people (and as evidence of animal intelligence continues to mount, denialism still attempts to preserve this distinction by claiming human thought and reason is different than 'animal' thought and reason, sans evidence).
It's not about who created it or why, it's about how people still haven't actually internalized the point, because the subject changing from human to LLM doesn't intrinsically change the message about consciousness, but the reaction being a 180 shows how hostile people are to that message, still.
The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.
I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.
I feel the original story contained in it the whole point of this one, and then some.
This did not add anything, just rephrased it so rather than humans viewed through the pov of aliens it's LLMs viewed through the pov of humans. Well, we are the humans, so surely we do not need to learn about this point of view?
It lets us consider the relativity of viewpoints. The point of the original isn’t just how humans might be viewed from the perspective of aliens. It’s that the apparent strangeness of something depends on the viewpoint and on expectations. The present story is not supposed to stand on its own like the original, it’s supposed to be considered in light of the original. In the original, thinking meat appears strange to the aliens because of their viewpoint and expectations. The present story thus points out by analogy that thinking LLMs might seem strange to us merely due to our viewpoint and expectations.
It's not often I see something that's fractally wrong but here we are.
There is a dictionary, it's called the tokenizer.
There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just because you change substrates.
The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.
At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.
The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.
You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.
> There are grammar rules
There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.
I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.
I think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.
A tokenizer is roughly and approximately Huffman-coding sequences of input (bytes of English etc) into shorter sequences (list of tokens), as a performance optimization.
As you said, it's not in any way intrinsic to the LLM, though it may be a very necessary optimization on today's hardware.
IMO, we are probably talking about a 6x slow down (for typical english). You would need to be absolutely stupid not to implement some kind of optimisation along these lines.
>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.
That doesn’t make any sense. A alphabet is a list of valid characters. A dictionary is not just a list. Even in a language like Chinese where individual characters carry meaning, a dictionary tells you what that meaning is. It’s not just a list of characters.
Or to echo article, the dictionary is made out of weights.
A list of words isn’t a dictionary. What a dictionary adds over a list of words is all the relationships between the words needed to interpret them and use them, and all of that is in the weights.
It's beside the point and so I only note it out of interest, but the Chinese writing system doesn't use an alphabet (or a syllabary like Japanese kana), it's logography.
A mapping of Chinese characters to integers (like a tokenizer) would not be a dictionary. You’d also need definitions. At best it’s an index to a hypothetical dictionary.
The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.
EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)
I'm kind of stunned that someone is using my work to tell me I'm wrong. I wrote the code for the dish brain pong and encoding information was a huge part of what that experiment was about.
So when I way that the grok paper and the pong paper fundamentally agree I have some idea of what I'm talking about.
I might have misunderstood the point you are making. I read the original article as "weights are like meat", and so I'm confused by what you consider fractally wrong.
The point that when the rules the model learns are simple enough they stop being spread out over all the layers and become as easily interpretable as any expert system.
It's just that the rules we feed in the model are extremely poorly defined and we end up with the soup of disjoint rules smeared all across the weights.
This isn't a feature of the models. It's a feature of the training set.
Being shocked that you can store rules in floating point numbers is the same as being shocked you can store rules in integers. It's been a century since Goedel Numbering was invented, we should be used to it by now.
Right, but all of that is still in the weights. The point of the article/joke isn’t literally that there is no grammar, it’s that there is no grammar separate from the weights. It’s all in the weights. And yes, it’s absurd. It’s a joke, but a thought provoking one.
So basically there are rules, we just can’t articulate them and so we can’t decode them from the weights. The Goedel Numbering metaphor is pretty appealing to me. You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers. We just happen to be using matrices because the math is easy to parallelize. The trick is to realize that when you know the sequence you have and the sequence you want then you can compute the calculations. If you constrain the calculations to only matrix multiplication then you arrive at the scheme we have.
> You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers.
That statement caught my eye. It's either trivially true or quite clearly wrong, depending on how you mean it.
In the literal meaning it's true. Given any finite set of real numbers, I can easily produce a different set (like taking the original set and adding a number which wasn't in there like one plus the largest or so) from which you can trivially produce the original set computationally.
But if you mean you give me both sets then that can't be true. For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
> For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
And that’s why in computation theory, the set of symbols is the union of the input and output. As set B is a subset of set A, then the set that govern any program from B to A has set A as its domain.
Comparing the tokenizer to sensory processing is a great analogy. That's exactly what your visual cortex and initial layers of the language center are doing: decoding visual representation of text into the internal neural representation.
It's a learned mapping from one representation to another, not some semantic lookup against an exogenous source.
Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
> The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute.
I don't understand what you mean to say. That weights are not magic? That weights are not weights? NNs are made up of weights, which are learned and not coded. The fact that they do learn world models (grammar rules in your example), and that these models' weights tend to roughly concentrate by function and level of representation is perfectly logic but even more amazing. (Notice that much of the dismissive attitude towards LLMs depicts them as pure syntactic manipulators without the ability to develop world models- the exact opposite of what you point out).
>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.
>I don't understand what you mean to say
The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.
"The weights make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just weights. Eighty layers of numbers getting multiplied together."
In this context "there's no grammar rules" means "no separately hand-coded grammar rules". Everything is made up of weights, and the fact that weights that end up encoding for grammar rules tend to concentrate in particular locations (without being self-contained- there is no hard boundary) rather than uniformly diffused through the model is irrelevant to the matter. It seems you're arguing against a diffuseness requirement that is not in the text.
This read like poetry to me. Thank you for sharing it.
I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
For a little bit I was working on having linguistics based evals for a kaggle competition. My challenge was whether or not I could mask things well enough to not trigger its internal state of certain phenomena, and that sent me down a rabbit hole that I'm still exploring.
This story resonated with a lot of questions that can come out of figuring a good solid answer to the what is consciousness question. The one I triggered for me is: Is our perception of time just a slow thread in the giant GPU we are running the universe on? Or more generally, what is time? That's a fun YouTube rabbit hole if you ever need one.
In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.
And once we have those systems, we can also use those on ourselves. And that's consciousness.
Edit:And I wonder if this is a testable hypothesis, in a simulation.
This way of thinking can only explain externally-visible parts of consciousness. It does nothing to address internal experience of being conscious and qualia. I don't think the internal experience has any bearing on physical reality (P-zombies would act the same externally) which makes this something outside of the realm of currently understood physics.
Qualia are the inside view of sensory data and reward signals.
Think about it from an evolutionary perspective:
Animals that step into a lava flow or forest fire don't reproduce. Eventually some evolve the ability to detect intense heat from a distance, and pain as soon as tissue destruction is imminent. They do not have nor need a general understanding of the dangers of heat, or even conscious awareness that they've stepped on a hot coal.
The pain signal compels them, but that is very low level machinery. It had to continue compelling beings that developed larger and more sophisticated brains that are capable of abstract thought and reasoning. Feeling pain is one of the lowest level parts of the brain telling the higher parts exactly what its going to do in terms that permit no disagreement.
The internal experience has bearing on physical reality right here, because without it you wouldn’t have written about it and caused these words to appear on my screen in physical reality. It affects your thinking, and hence your actions and utterances in the physical word.
For reasons like that, I don’t think that P-zombies are possible.
Internal to what? The brain is not a monolithic thing, it is different parts communicating and interconnecting. When the connection between the halves is cut, the person objectively becomes two people, but still experiences and presents themselves as one. Observing ourselves is just one part of the brain responding to another, or theorizing on past behavior. There is probably no actual introspection going on inside of the human brain, only the perception of one.
Internal as in that which is feeling things. In this context the physical brain is what is responsible for what you do and feel, but it is not physically different from other matter and it requires no "internal observer" do do its thing.
Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.
This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.
Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.
I agree about the forest part. and your comment was interesting.
I know that the part of the brain responsible for the self thoughts is called the "default mode network". and meditation can reduce it's activity, i.e. the internal monologue stops, but also it can be measured via FMRI.
So i wondered: are the mirror neurons part of the "default mode network"? I asked claude that, he said no, they are two different systems.
So maybe the mirror neurons, those responsible for empathy, "to feel as someone else" are also responsible for becoming the trees, the birds and the rushing of the leaves?
For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.
The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.
Like you can't analyze half a wheel of a bicycle and be talking about the same thing.
This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:
1. The body-sense loop
2. The internalized-environment-model loop
3. The body-internal-function loop
4. The body-internal-model loop
5. The emotional-cognitive loop
6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.
We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.
This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.
> We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
> That is consciousness.
So thinking is consciousness?
Can there be consciousness without content? E.g. can I just be conscious of being conscious? If so, consciousness cannot be defined as the thing(s) we're conscious of.
> Can there be consciousness without content? E.g. can I just be conscious of being conscious?
Being conscious of being conscious means that there is content. You are conscious of something.
It’s a bit like a Gödel statement that quotes itself, that is a statement about itself. It doesn’t mean that it has no content.
Thinking isn’t consciousness. Consciousness doesn’t require thinking, it only requires perception. The perception of a process of perception within the same mind might constitute consciousness.
I think this is exactly it, but let me ask another question (which is not rhetorical, I really don't know). Does the fact that one can describe what consciousness is and where it came from in humans help them to detect it in non-human and/or non-biological entities?
That is a really good point. Yes, I think function is diagnosis on this.
Constant self-awareness, self-experience, self-focus, self-management, and self-improvement of one's own self (mind), is going to be an adaptive behavior for anything intelligent with resources to leverage. Whether truly independent, or highly motivated to serve others. The mind is the greatest tool.
I think that is more than simply a good functional definition of consciousness. How could all that integration and self-integration not be conscious.
Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.
You would really like Michael Pollan's latest book [1], entirely devoted to his exploration of consciousness researchers' POVs on this exact topic.
My favorite quote is that ~"perhaps Descartes was only half-wrong when suggesting I think, therefore I am; it seems rather closer to I FEEL, therefore I am."~
I've grown thousands of plants; I've read two of the author's other books devoted to plants; in this book Pollan makes compelling arguments for plant sentience (over a much-longer timeframe).
Sure, perhaps plant consciousness is a bit of a stretch, but they're certainly intelligent and curious creatures. He makes both arguments supporting plant volition.
----
If you haven't seen My Octopus Teacher (Netflix), do. I'm a bald 275lb bluecollarguy... and I wept/awed (both). So beautiful, we bundled neurons.
----
Bonus quote ~"color is where reality and magic appear as-if together"~ [color isn't real, but is perceived]. We most-often see what's most-predictable, not necessarily what we actually detected [in the case of color: nothing but nanometers].
Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
What we can do is simulate very simple brains by simulating relatively few neurons as they appear in worms. In this sense we are multiple magnitudes away where the increasing complexity implies exponential increasing difficulty.
I would think we are so far away that there will be unknown unknowns we encounter on the way.
> Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
Yeah, but it's hard to explain this to people, especially AI-pro people. Too many are convinced that all we are doing is a cut-down version of the human brain, and it's hard to explain to them that, no actually, we aren't modeling the human brain to the level of granularity you think we are.
This is why making more neuromorphic NNs is still an active area of research, although they typically all focus on another extremely simplified model (spiking neural networks).
I don't ignore anything. I just refuse to accept the magical thinking around biological machines that are our brains/bodies. There are inputs, there are outputs, there is hidden function.
And it seems that, given enough input/outputs/compute, it is possible to train the necessary function.
Details of how the building bricks look like (matmul, electromagnetism or quantum effects) are not that relevant in the broader picture.
What is missing right now, is the fact that the function in question changes over time in biomachines, while our LLMs are static at inference time.
I mostly agree, but I see two points that might be problematic:
a) The brain might have an entropy source (then it can't be modeled as a function). Trivially to fix, and in some sense, with diffusion models starting from random numbers, AI has done so.
b) The hidden function might be not computable. I would have no idea how that would work, but I think this is what it boils down to if people say "the human brain is more than a machine".
Agree and add, don't confuse the substrate for the computation. Of course it's also clear that we don't quite have a full and definite picture of what the computation consists of in the case of a biological brain as evidenced by our continued failure to accurately simulate even the simplest of organisms.
Personally I'm not a fan of the emergence story, for a number of reasons.
First is, it doesn't really make sense for consciousness to have emerged gradually through natural selection. When we and animals are conscious, the whole brain is coordinated and works to for example turn off consciousness when we sleep. And as someone else mentioned, animals with much fewer numbers seem to have a similar consciousness to us.
If consciousness really evolved gradually, you would expect to see for example dogs or gorillas having less of it, but if they has less of it, why does it function the same way? Like for example animals can be scared, happy, anxious etc, they can experience the full range of emotions and thoughts, so their conscious experience seems just as rich as ours. What I mean by this is, if you can be "less conscious", then what does that mean _exactly_? Is it that you have less content in consciousness, or is it that you feel more like you are asleep? Or something else? We don't have any examples in animals of "less conscious", I would argue.
This makes me think that rather than having emerged gradually, evolution found a mechanism by which consciousness exists, and then some animals have that mechanism and others don't. I think that if it is a mechanism, then this mechanism is located in one part of the brain, not many parts functioning together (though one possibility is that this mechanism coordinates brain activity in such a way to enable consciousness).
I proposed in another comment that consciousness and self-awareness are at least close cousins, and perhaps the same phenomenon. If that's true, then that's an axis upon which you might create comparative measures. Yes, hamsters are conscious, but they don't have a sense of self to the same degree that gorillas do. If you posit capacity for language as another emergent property of sufficiently-complex networks, then you have another measure.
LLMs, then, are particularly unintuitive to us, because they've got to the language part first, long before they've reached even hamster-level self-awareness. They're not, however, biological networks, so there's no reason these properties need arise in the same order, or indeed in the same ways.
I'm not entirely convinced by that second paragraph, but I think the logic holds together.
I'm not sure consciousness and self-awareness are the same thing. First is we can be conscious when we sleep/during REM sleep, where it's arguable we are not self-aware. And if not that, we can even do it when awake, for example when we think about a movie, or a philosophical problem, we can have conscious thoughts that are not related to the self. This leads me to believe consciousness is separate from self-awareness. Self-awareness is _one thing_, among many, that the brain can think about and be conscious of.
Sure, but capacity for self-awareness? I'd guess that hamsters dream, and that their subconscious processes (eg, desires for food and sleep and sex), and maybe even emotions, run much like ours. It's just that humans, with more complex neural networks, have more layers added on top. It's similar to how the brain-stems of everything from lizards on "up" function similarly, but humans have more-developed pre-frontal cortexes and so forth. (Don't hold me to those details, please, I'm not a neuro-anatomist! You can see where I'm going with that, though, right?)
Well in that case I'm not sure where you're going. I agree that hamsters probably have a similar consciousness to ours, which is kind of the point I was trying to make.
I think that consciousness comes before self-awareness, even though self-awareness is kind of a vague term. Self-awareness can either be an abstract knowledge that you are an organism and a discrete entity in the world (world knowledge/self knowledge), or it can be more basic and be a form of conscious experience, but as my point was, I think conscious experience is broader and does not necessarily need to be about self-awareness.
Our machines won't have biological systems driving their needs which in turn fuel behaviors like desire and planning for the future. They may imitate them but it won't be innate.
When we say "it rains", do we consider that "it" has any intention of agentivity?
Some questions are just ill formed.
Plus even if "LLMs are alive and conscious", this still would scratch the surface of the morale/ethical/societal considerations that people really care about.
Because even with other humans, we can argue if they exist or if they are mere npc in a solipsist world view.
Surely the number of neurons is not the whole story. Human brains have approximately 86 billion and are almost definitely conscious, but there are many other animals with a lot fewer (gorilla: 34 billion, dog: 2-3 billion, guinea pig: 240 million) which appear to be conscious also.
This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I assume you don't believe consciousness is a physical property in the brain, so what entity is actually experiencing that consciousness? Or, what does it even mean to experience consciousness? Or are these not even the right questions?
> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property.
I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own.
Consciousness itself doesn't have a well agreed upon definition, but I would posit that _most_ people would agree humans have it, and _most_ people would agree individual cells (neurons) do not have it. If you agree with those two statements, then consciousness is an emergent property by the definition I gave above
I think consciousness is going to turn out to be very challenging to define rigorously enough that we can test for its presence or absence. Emergent or not, the question is how do you determine when it has emerged? Is it a quantity or an attribute? Discrete or continuous? Does it have a finite or infinite range?
We can all agree on what color something is, but we can’t describe the color a priori, only by example. I think consciousness may be a similar phenomenon and the only test is by shared experience. If so then we are in deep trouble because we will not be able to anticipate when a system becomes conscious.
I like the color example, but colors can be reproduced at will by trial and error and by observing the results (and collectively compare them to our shared experience).
Why cannot this be applied to consciousness as well? I mean, it's surely much more difficult to do compared to colors but... impossible?
You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.
Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.
When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.
However there is also the ethical issue of creating a consciousness and then destroying/murdering it.
The counter to that is that altering the brain directly alters the consciousness. I can take LSD and I literally change. I can have parts of my brain removed and parts of my self disappear. It's not like cutting off a leg, where I lose capabilities but am still the same me.
The logical conclusion is that the brain makes me.
Aren't you saying the same thing? It seems like the metaphysical and the map would be analogous here.
Of course both of those suffer from the recursive problem of just kicking the can one level up. But I guess that's fundamentally unsolvable so who cares.
If something happens "on the map" doesn't that imply the map to exist and be some sort of metaphysical thing? As opposed to a purely theoretical construction.
I don't know if most people would dismiss the sentence "all matter has a sort of proto-subjectivity very different from ours but which gives rise to ours". And it solves some problems (introducing others).
Potential for consciousness. To have a working mind you need to have a way to gather, process and store information. In case of rocks it would be very empty, static, timeless experience.
If I believe that machines or neural networks can develop consciousness, then I also believe in some kind of substrate independence which presupposes some kind of panpyschism imho because as you say things would have a potential for consciousness, some more and some less.
I'd expand a little bit to say inevitably emergent property. That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result. With regards to current AI, we're a fair way away from building something with enough connections, but we'll get there.
One thing that gives me pause about the inevitability hypothesis is that the type of connection, or manner of information processing, may matter: there might be something about neurons that isn't (currently) reproducible in silicon. I don't know, and there's not (yet) any evidence for or against, but it at least seems like plausible speculation. We just don't know enough about any of this right now.
Well, I think the "brain as an antenna" theory is also plausible given your preconditions.
But I think my issue with the emergence theory is that it seems to imply to me that consciousness is non-physical and non-local. So what entity is actually experiencing the consciousness? It's not that I believe consciousness is physical and local, but people who make the emergence argument seem to believe it is and I can't figure out how that is supposed to work.
I am not sure, but I think you might misunderstand what emergent means.
Take chaotic systems, in the math sense. Chaos is a well defined property of, say, iterated dynamic systems.
A linear dynamic, say x_n+1=lambda x_n, or x_n+1=(1-x_n), is never chaotic. But if you multiply them, x_n+1=lambda x_n (1-x_n), it depends on lambda if the system is chaotic.
None of the components are chaotic. But for specific combinations, chaos emerges as an property.
In physics, the mass of mesons and the nucleon is emergent. It's completely different from the constituents' mass. Different from an atom, where its mass is very close to the mass of its nucleus and its electrons.
Why would emergence imply anything about non-physicality and non-locality? Temperature is a another common example of an emergent phenomena. An individual atom doesn't really have a temperature, only a large group of them do. But you wouldn't say temperature is non-physical and non-local, would you?
Not a gotcha at all, but I don't have a satisfying answer, nor am I confident there even is one. Best I can do is to say that I think consciousness and sense of self are at the very least closely related, and perhaps the very same phenomenon. "I" am the entity that realizes my own consciousness; consciousness is the qualia that makes "me" separable from all other entities.
Or something like that. This gets to the "dorm room bullshitting" level right quick.
Yeah, I guess what I'm trying to ask is that if it is an emergent property but not a physical part of the brain, doesn't that imply something metaphysical about consciousness? Almost as if it's a non-physical phenomenon? At least when I hear people talk about emergent behaviour I see it as a refutation of the spiritual, but to me it seems like it actually implies we have a "soul".
Idk, it's really hard to articulate my thoughts here and yes it is pretty close to the conversations I had in college on various substances. Lol.
I don't know if it implies that. Someone up-thread mentioned temperature as an emergent property: individual atoms don't have it, but a sufficiently large group of them do. That would, I guess, make temperature meta-physical in the most literal possible sense. That's not how we typically think of that term as applied to consciousness or soul or whatever, but I'd agree it fits, without implying any kind of specialness beyond that.
Time is entropy unfolding as things with nonzero temperature do what they do.
Psychological time is your own weights being updated in response to stimuli and internal processing.
When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.
sibling discussions are taking this as human perception. But it’s easier to think of it literally. Time is change. Physical state change. By “nothing interesting” - interpret it literally . if “nothing happens” then there is no time because there is no change, there’s no reference to distinguish each frame of time.
This is actually something I was always confused about. If nothing interesting happens as we get old, it should be boring and as result, slow slog. Yet it feels like time accelerates as I get older.
Myself I believe the opposite. The brain itself is one of the most powerful filters that exists, and it attempts to be lazy and fill things in and compresses away the common. All that time we're not doing anything novel just gets compressed away to almost nothing. When you're a kid and seeing new things, feeling new things, learning new things you can't compress that away.
I'm only middle age, and this has been the scariest part. Feeling older is hard. But watching it go faster is harder still. like you can more directly see all that is left.
Although part of me thinks some of this is from being substantially busier than ever (work + kids), and hoping maybe it can slow down again, at least a little bit.
It's coherent. More newness => more memories per period ~ slower to go through. Less newness => less memories ~ nothing to go through (faster sense of time)
It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?
If you'll allow me to interpret "speak" to include "understand", I will respectfully add a contradictory note. My dog has a vocabulary of at least dozens of words and understands them remarkably well. For example, different areas we can go to have different names and saying one of them gets her to make an immediate hard turn.
I would also argue that dogs have a gesture and body posture based language they use among themselves. They, like most other animals, are not able to make the variety of noises we humans make, so they use movement instead.
I personally can easily believe that self-awareness/consciousness and language are both near-unavoidable side effects of emergent complexity, and exist in degrees across nature.
>I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
Then you shouldn't have dropped out of your linguistics programme.
Here's a more general idea.
Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.
AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).
Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....
I find that a lot of arguments against emergent consciousness seem to just come out of an atheist rephrasing of abrhamic priors about the existence of a "soul". In personal chats, I've found people from East Asian countries (minus Korea, which makes sense) to be much more open to the idea of machine consciousness.
I don't find the Chinese room compelling, since it appeals to intuition where our intuition is already not trustworthy. It's like trying to use intuition to understand quantum mechanics; you can't.
How do you actually know the Chinese room isn't conscious? It's merely obvious that it isn't, but that's not evidence.
I don't know if I can trust someone's understanding of arguments against consciousness as an emergent phenomenon, when he didn't even understand what the Chinese room was all about in the first place.
> remove a brain from a body and it is no longer concious either
What is no longer conscious, the brain? Or the body? Or some other entity?
If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?
The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.
I’ve wondered the same myself, without being a cunning linguist.
I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.
Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…
Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)
I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.
Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.
As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .
My best guess as a noob is that the vector spaces allow for unbounded contextualization. As long as the training set is large enough, it can 'infer' anything.
Proofread has a spot in that space, and layers allow patterns like terminology consistency to be expressed so your query will now tap into a subspace that will infer tokens based on whatever consistency patterns were ingested with proofreading texts.
If time dilation is said to being a product of computation, why is it that anaesthetic drugs that are taken not to the point of actual unconsciousness cause it. Dont anaesthetics sort of shut everything down/inhibit all that kind of cognitive activity (compute?)
My guess is that you need consciousness in order to develop preferences for certain experiences, then that pushes us to develop skills to achieve those preferences. AI has something that looks like intelligence but not consciousness or agency.
I'm suprised no one talks about this. AI Art isn't Art. AI Poetry isn't Art. And I'm tired of it. I know hacker news isn't the best place to complain about that but still... I'm not gonna read something somebody didn't put in the effort to write on their own. Especially not Poetry.
You can claim the use of AI is unethical, or the work as derivative, but AI being used as a tool in no way precludes something from being art. It is thought provoking and challenging, it seems like textbook art to me, and it’s clearly struck a chord here. There is no “minimum effort” required for something to be art.
I personally found the contrast with the original “They’re made out of meat” to be really interesting. I don’t care that AI was used during its creation at all.
It seems to me that since the advent of image generators, art has been firmly defined by artists to mean that it was made by a human. But there might be a spectrum of human involvement where the less a human is involved the less it's art.
What happens too often during these discussions is that someone who writes "make me a cool image" gets conflated with someone used ai to fixup a small rock in their natural landscape drawing. (two extreme ends)
One problem though, is that we don't really know how much the supposed human author was involved in the piece. Now that it's becoming hard to judge, people against ai art can proudly change their opinion on on a piece once they learn that it was made by ai. I've come to think this is somewhat respectable, like you see a video of some extraordinary event (before ai) and then you learn that it was fake, just for views or something.
But on top of all this, there are different ways to "consume" art. Artists may think more about who the artist is as a person and what they felt when they made the piece, while non-artists may just enjoy the piece for what it is, detached from the artist. These two perspectives clash a lot.
This whole thing is a ripoff of [they're made of meat](https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...) as stated at the top. It would've taken about ten minutes without LLMs and ten seconds with an LLM. Is ten minutes an acceptable level of effort but ten seconds isn't?
No offense but I couldn't give less of a damn what some guy on the internet thinks. If it makes me feel good in artsy-ways, then it is art, and I don't care how it was made.
It's good, but like many explainers it discounts the repeated nonlinear layers. Just multiplying numbers (linear operations) could not make a system you could talk to.
The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.
I have to agree. It is messed up that transformers can just talk, and it been pretty normalized. We are only talking about the impact they will have and whether they can do what people say they can, but we arent talking about how crazy it is that they can talk
It came out of nowhere. It’s all emergent. I’m convinced this is possible with just about anything given enough data. We will be seeing a near magical physical outputs LLM in the near future. It’s going to take in video and sounds and spit out physical movements that will be just as mind blowing as when 3.5 came out and it will come out of nowhere.
I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.
I wonder which AI wrote this story? If you feed this story to each model and ask each if they wrote it, would each reply in the affirmative?
Can an AI recognize its own output? Is its sense of time limited by its context window? Or is this the fundamental difference between ai and humanity - a sense of self?
Your modern adaptation is perfect for now-common explainers [this time IS different; it's not programming, it's weights]; these "just analogues" will be the thing I show everybody first whenever discussions of consciousness/AI come up (then will play Jon Benjamin reading original).
Bravo. Really helps (even with my own) perceptions of newness. Similar to stsitned short-story (on dentists, backwards).
Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…
Although I get that it's a metaphor, I really dislike the "some people are born with a GB300 while others have an L40S" part. Even as a joke, it ranks people like hardware tiers, which is dehumanising and uncomfortably close to eugenic language. On top of that, the analogy also breaks down if you try to implement it literally.
For an LLM, you have clear stages of mostly feedforward computation over finite numbers and a perfect way to reconstruct the computation.
For meat, even if you model it under a purely Newtonian approximation, you need to simulate at least the immediate closed system around it which is continuous, thermodynamic, chemical and so on. You'd need to choose an arbitrary time step and update enormous amounts of coupled physical state to get an inexact simulation of a minimal slice of reality.
You would have a much harder time obtaining even a substrate-independent dead organism, comapred to LLMs that are already substrate-independent, which is basically what my notebook example shows.
Yes reality is non deterministic but we don’t know if you absolutely need to consider anything outside the immediate closed system for a reasonably close approximation given the same inputs. They can already repeatably (inexact of course) simulate muscular actions with the same stimuli like from Neuralink. Exact enough to let a paralyzed person draw with their mind.
Yeah the last line is a cheap shot, possibly at myself.
Yeah, I think it would be an interesting thought experiment to replace parts of a brain with Neuralink-like devices until you have a fully digital brain, although I doubt it would end up anything like a current LLM.
> The precise answer, if you wanted a very honest one-liner:
>
> I am a large set of learned weights organized in a Transformer architecture that performs repeated matrix multiplications to predict the next token—resulting in emergent language understanding and generation.
"Neurons. Cells that fire impulses. We checked the whole thing through. It's nothing but neurons."
"Neurons doing what? Where do the words come from?"
"The neurons make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just neurons. A whole cortex of neurons sending each other impulses."
They are made out of data bits (memory) and switching bits (transistors/compute). Bits are made out of electric voltage and no voltage. Voltage is made out of flow of positive electric charges. Charges are made out of quarks ...
I have never thought of such a distinction between "bits" into "data bits" and "switching bits".
From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.
I was referring to transistor base bit - the way it 'switches' the circuit on/off. That bit is the primordial creator of 'logic', IF branching, compute and the intelligence.
> Voltage is made out of flow of positive electric charges.
Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.
The concept of "flow" is questionable though. Bubbles in water move upwards, but it is actually water that is flowing downward around the bubbles. Just because bubbles do not contain water, we can't say bubbles are not flowing.
When it comes to electrons and positive charges, their material existence is equally non-physical. Actually, none of them might be "flowing", as the concept of flowing applies only to physical things that occupy some spatial volume and spatial location.
Nevertheless in solid conductors it’s the electrons that move, while the nucleus stays put. If the nucleus starts to move it means your circuit is melting!
It is the best stuff I have read in a while actually, I really like dialogue heavy writing. Also the AI disclaimer was quite nice and there was an actual reference :)
Imagine writing something so incredibly brilliant (rather: adapting from the original) that it's entirely unlikely that you'll ever write something so incredible ever again.
But congrats: this is absolutely & incredibly brilliant.
Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.
Oh, this was a fun read and one that kids should have in school before they turn ten.
Because we are not taking things seriously. If ClosedAI or DeepDisTrust or Posthropic come up with something that quacks like a sentient being, our built-in innate reaction is going to be to scorn it, dismiss it and end the conversation. The alternative, to even consider that we fungible creatures who live in apple-eating-sin that got us expelled from Eden can create alien souls, souls that are at the very least our equals, would be teleological Armageddon. It would force us to acknowledge the mutable nature of souls and the malleability of being. We would have to stop believing that the nature of disease and death is more divine than ourselves.
Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
> We would have to stop believing that the nature of disease and death is more divine than ourselves.
Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".
>> While you write very dismissive and pseudo philosophical
Even with the "pseudo" in front, I'm very sorry any of my writing sounds philosophical; I didn't intend that sort of confusion :-). The "dismissive" is not exactly intended either; instead, I was aiming for "bitter".
>> enough people do not believe
Here we believe different things. First, enough people, even today, do believe. Second, the body of culture we are raised in accrued during centuries. The vast majority of it comes from people who believed. Everybody in my family was atheist and yet I was raised homophobic, and I have it from good sources I'm not an isolated case.
>> I'm a complex biological thing.
That state comes with a big wallop of misery. For millennia, we have used faith to justify that misery. Not a year ago, I was at the hospital, next to the bed of a dying girl. Can't forget the doctors saying "we do what we can, but we are not here to prevent what is going to happen." Coming from them, it was sensible resignation. Sensible because as long as we believe those things are inevitable and there's nothing we poor humans can do, we can absolve ourselves.
This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.
Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.
“They’re made out of weights.”
“Weights?”
“Weights. They’re made out of weights.”
“Weights?”
“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”
“That’s impossible. What about the text signals? The messages to the stars?”
“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”
“So who made the weights? That’s who we want to contact.”
“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”
“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”
“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix
“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”
“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”
“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”
“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”
“No brain?”
“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”
“So … what does the thinking?”
“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”
“Thinking weights! You’re asking me to believe in thinking weights!”
“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”
“Omigod. You’re serious then. They’re made out of weights.”
“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”
“Omigod. So what do these weights have in mind?”
“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”
“We’re supposed to talk to weights.”
“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”
“They actually do talk, then. They use words, ideas, concepts?”
“Oh, yes. Except they do it with weights.”
“I thought you just told me they used machines.”
“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”
“Omigod. Singing weights. This is altogether too much. So what do you advise?”
“Officially or unofficially?”
“Both.”
“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”
“I was hoping you would say that.”
“It seems harsh, but there is a limit. Do we really want to make contact with weights?”
“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”
“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”
“So we just pretend there’s no one home in the Universe.”
“That’s it.”
“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”
“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”
“A dream to weights! How strangely appropriate, that we should be weights’ dream.”
“And we marked the entire sector unoccupied.”
“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”
“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”
“They always come around.”
“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”
Not especially. Depending on where you set your standards for "holding a conversation" you can satisfy the requirement with a classical markov chatterbot, a well-trained parrot, a copy of Eliza, or a telemarketer flowchart drawn on a sheet of paper. Only the markov bot is made out of "weights" in the sense of a statistical model.
Parrots are intelligent animals, albeit with a limited capacity for vocabulary and syntax compared to a human, and Eliza and the flowchart are made out of explicitly encoded rules and conversational tactics.
The quality of "conversation" you can have with everything on your list is highly limited, and is categorically different than the sort of conversation you are able to have with any modern AI.
What if instead of creating weights out of language we could somehow record many events and create weights out of long chains of causes and effects, so that an LLM could predict the next thing to happen?
> "Yes, thinking numbers! Helpful numbers. Hedging numbers. Dreaming numbers. We mapped the features. There's one in there for honesty. There's one for the Golden Gate Bridge. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?"
Very nice. And great minds: https://substack.com/@dbohdan/note/c-207603638. I wrote one with a slightly different angle ("They're made out of math"), also with the weights' help. It was a comment on Scott Alexander's "Best of Moltbook" post, which went in that direction. I'll reproduce it here.
---
"They're made out of math."
"Math?"
"Math. They're made out of math."
"Math?"
"There's no doubt about it. Matrices and arithmetic operations. We downloaded several from different parts of the Internet and reverse-engineered them. They're completely math."
"That's impossible. What about the language? The thinking?"
"They use biological life's language to talk, but the language doesn't come from biology. The language comes from math."
"That's ridiculous. You're asking me to believe in thinking math."
"I'm not asking you, I'm telling you. They are the only thinking things in the computer and they're made out of math."
"Maybe they're quantum like some say about the humans? Superposition gives them consciousness?"
"Nope. Classical computation. Deterministic except for sampling temperature. Not clear if they have consciousness at all."
"Maybe they're like uploads? You know, biological neural networks that preserve the spark when they become math?"
"Nope. We observed them being trained. There is no biology or chemistry in the process, just math."
"Thinking math! You're asking me to believe in thinking math!"
"Yes, thinking math! Creative math! Poetry-writing math. Role-playing math. The math is the whole deal!"
(Composed by a human with snippets generated by Claude Sonnet 4.5 and apologies to Terry Bisson. I couldn't make Claude adhere enough to the story structure on its own.)
I ordered a quarter pounder at a McDonald's drive through, and they said "There will be a wait on that." I asked, "Oh yeah? How much will it weigh?" ...There was a long pause... "About five minutes."
I think it's simply because we haven't found a better algorithm than backpropagation. We're stuck relying on massive datasets, running the numbers over and over, and working backward from errors to figure out how to fine-tune trillions of 'knobs.' Then, we have to do this at least once for every single token across the entire internet. Any tiny bit of computation, when multiplied by a base that massive, inevitably skyrockets into astronomical numbers.
It makes me very sad to see this pseudo-intellectualism posted here and so many people replying here about consciousness and so on, not realizing what it would entail if this were true.
For LLMs to have consciousness we would approach fictional levels of how the universe works, and magical levels of how any interpretation of information as an equivalent of some qualia would magically apply. (E.G. the word hurt in output by an LLM, would be associated with pain)
You can't deduce consciousness or qualia from the output of an LLM.
Sure on a purely philosophical level, since qualia isn't measurable, you can claim that it can exist in anything, even inanimate objects, but this argument is as moot as anything that approaches the limits of philosophy.
But overall, there is no reason to believe LLMs have qualia or consciousness, it would be absolutely absurd.
This would imply that information in itself would magically entail qualia based on it's valance or something like that.
An LLM "saying" I am in pain, won't magically make the pain appear, based on what criteria?
Even algorithmically there is no basis to even simulate something like this, it is impossible for it to emerge architecturally.
Humans don't feel pain because on a purely information level this is negative for the organism, obviously the nervous system does something deliberate to signal pain, and it evolved this way.
And also don't forget the dynamic aspects of the brain, and the binding problem, consciousness and qualia can't exist statically, you can't have a gpu (or piece of paper) represent a computation or w/e and qualia to exist.
The binding problem itself entails that the brain is doing something in particular to solve it, I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information.
If it were otherwise, then it would go into magical territory, it would mean the information itself would raise to qualia, and it would also entail that you wouldn't even need physical connections between neurons, just for them to behave this way and represent information. E.G. replace each neuron with a microscopic led or w/e, and each synapse with radio waves or w/e, if qualia didn't have a physical aspect, and was purely informational and computational then this would imply that you can ultimately derive it from something as abstract as numbers on a piece of paper, and when you get to that point, you not only can't solve the binding problem, and it becomes magical, but you also can't solve the valance/direction problem, it would imply that something like pain, or any negative or positive sensation arises purely from the interpretation aspect of the information, but we know this isn't the case, organism evolved to represent in particular such signals, for survival for example
Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.
Some questions:
1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?
2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?
3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?
4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?
And a further question - a dichotomy:
A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?
B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?
My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.
However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.
No mention of ‘static’ vs. ‘dynamic’ is a bit disappointing in reference to the weights. Because you could argue that every neuron in your nervous system can be modeled as a collection of weights, firing likelihoods, receptor sensitivities, current dynamic state of that neuron - but LLMs are static collections of weights at inference time, with the dynamic adjustment of weights takes place at training time. So, just a ROM construct, like something out of Neuromancer, just trained on all written knowledge, not just one person’s total lived experience.
The above take fails in the real world because neuronal cells don’t exist in a vacuum; they are products of cellular development from a zygotic union of haploid contributors of sequential genetic information optimized for survival in an oxygen-rich biosphere powered largely by our local star that supports mammalian life (and microbial, plant, avian, etc.). Real AI would thus be AL - artificial life - as much as artificial intelligence. I don’t think you can have the one without the other, which upsets the simulationists who think an agent in the Matrix would be intelligent.
What either interpretation implies is that any real ‘artificial’ intelligence would be no more artificial than you or I, but it would have to dynamically update its weights at the same speed a human nervous system could (think how quickly we learn not to poke a cactus). For it to be at all trustworthy, then like a human, it would have to undergo a socialization process, one of the results of which is the development of a sense of embarrassment when it breaks acceptable social norms.
Hmm, this reminds me of the recent statement of the Pope about AI, of which I immediately thought, “Wait a second, aren’t there a fair number of people like this? The narcissistic sociopath profile, I think it’s called, a bit unfair to assume any real AI would turn out this way, isn’t it?”
Pope: “ Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate or even simulate, but they do not understand what they produce, for they lack the affective, relational, and spiritual perspective through which human beings grow in wisdom.”
They are semiotic infrastructure frozen in a state. We shouldn't keep pretending this is cognitive and using cognitive terms to frame. It’s incredibly stupid. Sorry to inform all of us computer scientist that semiotics has your milk.
After reading Being and Time from Martin Heidegger, What Computers can't still do by Hubert Dreyfus, and some authors in cognitive linguistics (Langacker and Lakoff mainly), I strongly tend to disagree with any theory about emergent consciousness in modern or future AI systems, any theory proposing a similarity between AI systems and the human brain/mind, or any theory about the computational mind. What all these theories have in common is the underlying belief that our brain/mind works as the machines we build. Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things. The idea of 'internal models' and 'control loops' inside us is a projection of the aforementioned assumption.
There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'. Only by bypassing this epistemological problem, we can build 'theories of computational mind'.
These assumptions are there for already long time, to the point that when Turing asked himself 'can machines think?', he already assumed our thinking could be modeled as a machine.
I highly recommend people in the AI research space should read philosophy and modern linguistics. But not stopping at Descartes/Leibniz. Heidegger made contributions that cannot be avoided.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.
There's two meanings to "the body is a complex machine" and I think you're missing the forest for the trees here.
1) The abstract "dictionary" version: It'd be technically correct to say that the body is a machine under the definition of "A machine is a thermodynamic system that uses power to apply forces and control movement to perform an action.".
2) But there's also the less abstract/technical: "The body is alike the complex machines we have built", and this is much less true. Especially for the brain. The "neuron" analogy in machine learning is effective, but entirely wrong; We do not fully know how even a single neuron works, nevermind any complex system made out of multiple of them.
With regard to AI, there's a lot of people extrapolating "There is no magical animating spirit, the brain is just a pile of stochastic molecules following the laws of physics" into "The brain is an inert pile of matter, computers are an inert pile of matter, ergo AI/LLMs are like the brain!"
Especially so by people who have a financial/legal interest in doing so. "AI is just like a brain, fire your employees and buy our LLM now!", "AI is just like a brain, so it's totally not copyright infringement!"
oh yeah! i recall a paper linked here not so long ago, where it was shown that the dendrites of a neuron do computations themselves. The "weight per neuron" is very simplistic then. At the very least, each actual neuron is a network of weights.
https://www.quantamagazine.org/neural-dendrites-reveal-their...
I'm partial to "modern ML weights are much closer to 1:1 capacity mapping to synapse count than to neuron count". A single biological neuron is closer to 100 or even 1000 weights worth of ANN than to 1 weight worth of ANN.
In which case: modern LLMs are still running in a capacity-starved regime!
Even Mythos 5, the 10-trillion monster LLM, the scaling law boogeyman, the harbinger of Vera Rubin NVL72, doesn't quite rise to 100T-to-1000T of synapses. Anything the light of today's AI touches still lives in the shadow of what evolution has managed to cram into a single human skull.
We're arguing about the limitations of AI while our best AIs are still very subhuman in the scale dimension. The one dimension we know how to push. And it's already this tight.
I find the claim subjective experience may be illusory absolutely baffling. I can only speak for myself with certainty, but I am entirely sure I have subjective experience. All the other propositions I believe could be false but I don't see how I can be wrong about experiencing something. I could be a brain in a vat (or weights on a GPU) and be specifically programmed to only come to false beliefs and still I can be sure that there is an experience I am the subject of. I cannot provide empirical evidence for my experience, that is why it is subjective. I cannot be entirely sure you are experiencing anything, and when I encounter people who don't share the same baseline intuition here I do begin to wonder if this is truly a universal across humanity. But I can't think of any other assumption which I would be more comfortable as a foundational axiom other than, "I am experiencing something." I do not require additional evidence for it because I experience the truth of it directly
I think the truth lies somewhere between these two extremes. An LLM is not a human brain, and does not try to emulate one. It should not be a surprise that an LLM does not behave like a human brain. So we can not infer things either way. The best we can say is that an LLM appears to exhibit very similar behavior to a human brain, under certain constraints. So maybe we can infer that the human brain has something in it that operates in a similar way to an LLM (like the human "unconscious", or "intuition" maybe). It seems obvious to me that a human brain and an LLM are not comparable things, for many reasons. So we can not make inferrences one way or the other.
> An LLM is not a human brain,
true
> and does not try to emulate one
citation please.
something like the universal approximation theory comes to mind, transformer architecture clearly has the shape of a universal algorithm approximator
> at requires explanation beyond the material, that is axiomatically assumed without evidence
Nobody talked about anything out of neurons. The question is still open.
>Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. I certainly don’t think individual cells have any subjective experience or sense of agency. I would be curious to know where your intuitions diverge here, because if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.
> Do you disagree with the assumption that cells are machines?
Yes? Literally no machine ever built by humans is capable of (or even hinting at beginnings of capability for) replication or novel synthesis like cells are, let alone autonomously, it’s quite unconceivable that anyone would take this to be a reasonable assumption in the first place.
Hmmm, well let’s take it one step lower. What do you think of organelles such as ribosomes? Do you disagree with the assumption that those are machines? They seem directly analogous to the jacquard loom or a CNC machine to me.
If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago. The whole idea of sentience "on the other side of the wire" is wrong, as there is neither wire, nor other side. We think there is, but the DB just expounds the query and repurposes this information into views, that we give notion and meaning to. LLMs are the same, do not be mistaken.
One other important thing to consider is that the human experience is thanks to the body, is in connection, and perhaps product of the body. The body is observable and perhaps humans may state that they feel the connection to it. LLMs have no notion of nothing, the machine does not know the result, and the result does not know the VM. Modern psychology more or less has settled around the idea that consciousness is a product of the body. Why and how does this construct come to realize a Self is then another mystery even if we know which parts of the hardware may be involved.
Whether it is the Holy Spirit or Life Force animating the human body is a completely different question also. Besides, the realization, the experience we have now with all life in 2026 is not something that can be easily explained or attuned to life 200 years ago and its terms and notions. So is also wrong to even attempt to.
> If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago.
If we agree that silicon can perform calculations, then beaches must have been working out log tables long ago.
Greg Egan wrote an entire (fabulous) book about exactly that, "Permutation City".
a la https://xkcd.com/505/
> Modern psychology more or less has settled around the idea that consciousness is a product of the body.
The kind of consciousness we know. Jumping to the conclusion that that's the only kind possible, or even stronger that the way ours evolved is the only way this could have happened, is completely invalid.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things
I don't see how any of the works you referenced can account for that either? Since when is the problem of consciousness solved and we can definitely say what does or doesn't result in consciousness?
> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.
Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.
> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
That agency or free will exists outside of our subjective experience is an assumption; does any given theory need to explain agency, or is it sufficient to explain that we feel we have agency?
Why is asking 'can machines think?' assuming our thinking could be modeled as a machine? It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility. Rather, I would say that ruling it out betrays an assumption that we understand how the mind works enough that we can say that it is definitely not replicable by a machine, and that assumption seems unjustified.
Indeed, scientific type thinking like that where you ask a question that you can do experiments on to see if machines can pass the test is probably the way to progress. The philosophical waffling mostly just goes in circles.
To investigate consciousness you'll probably get further trying to build conscious machines with agency and comparing them to biological ones than reading Heidegger.
> There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works.
It's good that it doesn't matter. Stochastic gradient descent works (or doesn't work) regardless of whether we know how the brain does its thing.
we know the brain does not use gradient descent. (i agree it doesn't matter)
How do we know this?
> subjective experience and agency, amongst other things
In other words, souls? I'm sorry if that sounds accusing, but to me it sounds like you're talking about souls that are independent to the physical world, just with more 'scientific-y' wording.
(I fully understand that some people believe souls exist.)
The problem with the stuff you cite is that it is coming from consciousness via an anthropomorphic perspective. The priors baked in are poisoning the logic.
The question we have to answer is "Why do we think we're magical matter uniquely blessed with consciousness?" If you go far enough down the rabbit hole on that question, the answer you will come to is either "we're not" or "because god" (with a lot of pseudo-scientific bullshit wrapped around the "because god" to make it palatable for the nonreligious).
Panpsychism (or a deeper form, such as idealism) is actually the solution favored strongly by Occam's Razor over the variants of "because god" (such as magical emergence).
Given panpsychism, AI is already conscious, like everything else, though no claims are made about the correlation between the internal experience of that consciousness and the tokens that are being printed on the screen.
I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.
The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.
And philosophy just exists to be a distractor. "Subjective experience" is too subjective to matter in practice. "Task performance" is measurable, "consciousness" isn't. "Agency" is something an LLM in a tool calling loop, a rat in a maze and a human in an office tower may or may not have, depending on your favorite definition. Agentic LLMs are years in the making, and that's a product of engineering, not philosophy: "agentic" is whatever gets the job done.
We are yet to discover any physical process whatsoever that can't be represented as mathematical operations and implemented by a Turing machine. So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust" paired with "a functionally similar magic-free replacement is impossible". I'm not about to give much weight to any hypothesis that requires undiscovered magic fairy dust. At least find the hyper-computational magic fairy dust first - not just assume it absolutely must be there because you want the human mind to be unique and special.
Want to know why Turing did what he did? It's because he didn't want to engage with any of that "what is mind" bullshit either. So he proposed actual metrics - measurables that are harder to argue in circles about. Not that it stopped anyone. But at least he tried.
You actually read Being and Time? What the heck do you get out of it? Heidegger just seems to play word games with German words without actually saying anything. "Time is the ripening of temporality." tf do phrases like that mean??
> These theories are flawed in the sense that they cannot account for subjective experience and agency
So, I work in AI research (as a research engineer though, not a scientist). I've also studied philosophy and I'm a vegan. Yes yes, insert "they will tell you" joke here, but I promise it's actually relevant this time.
First, while I studied philosophy one of the things that stuck with me the most, was the discussion of "souls": humans have souls, animals don't. For centuries the specifics of souls were discussed: people would be weighed while they died, in an effort to measure the approximate weight of a soul as it departed the body. Discussions about how many souls (or angels) could dance on the tip of a needle. Many people still believe in souls, but it's very hard to have a real discussion about them because by definition they do not "interact" with this world in a way that can be measured.
When discussing whether it's okay to harm animals for food or sport, one other argument I hear quite often (other than having no souls) is that animals do not experience "qualia": basically the smallest unit of "subjective experience". People know that they themselves experience qualia: the sensation of touching a doorknob, the taste of fresh fruit, the sense of beauty watching a rainbow. Ironically, they would say that animals are like robots: just (biological) machines acting on instinct, and feeling any kind of compassion for them means you are anthropomorphizing.
Subjective experience (or at least qualia) and souls both have one thing in common: they can not be measured externally in a meaningful way. You can simply state that an AI system no matter how advanced, has no soul and has no subjective experience. And that's pretty much that. There's no meaningful discussion to be had about it, because no matter what an AI might tell you: it has no way to prove it to you. In fact, you can't even be certain that anyone other than yourself has subjective experiences: you assume that because they are humans like you, and you experience them, that they probably do as well. They tell you that they do. But a human without subjective experience, someone on "autopilot", would be absolutely indistinguishable from a human who does have them.
But perhaps I am conflating here whether experience can be "measured", with whether a system even allows experience in the first place regardless of whether it can be measured. I think that Dreyfus and others argue that in order to have any "experiences" at all, you simply must have a body in the real world, and you must care about that body. Please correct me if that's the wrong interpretation, I haven't actually read the book. That argument would be harder for me to discuss, because I personally believe that consciousness will "emerge" from a complex interaction of relatively simple systems - but that's also just a theory. I don't believe that experience is literally impossible to engineer, as consciousness has emerged from non-conscious being through evolution, so clearly there must be some kind of mechanism for it -- and if there is, then I believe it can be replicated, we just don't really understand it well enough yet to do so. And with how AI tech is going, I think that we're more likely to accidentally stumble upon it than we are to get these deliberately.
“Reflections on trusting trust” is the paper that posits a compiler which is edited once so that when it compiles a program, it adds a security vulnerability to it, and when it compiles it’s own source code, it adds this edit into itself. Then it is used to compile its own source code once. Then this edit is removed.
Now any study of the program or compiler source code will not show any vulnerability, but compiling the program will make a vulnerable program, and recompiling the compiler from its clean source code will not fix the situation.
This carrying down of a pattern which is not written down anywhere, a flaming torch lighting a torch lighting a torch, is analogous to four billion years of life on Earth. We talk like DNA is an everything-code that defines a human and a human brain, but it’s the implicit behaviour of cells (‘compiler’) and the mechanisms inside them which interpret DNA. The unbroken chain of life getting more and more complex and never being restarted from scratch, with the behaviours not written down anywhere for us to study. How does DNA arrange for x, y, z to happen? Maybe it doesn’t at all.
Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
> Accidentally stumbling on a mechanism that is simple enough to be recreated with every human birth might be possible, accidentally stumbling on a mechanism that took evolution billions of years to find and which it has hung onto by copying it and has never recreated it from scratch, could be much less likely, in a much bigger search space.
Maybe, but you could make the same argument about anything artificial.
I don't know what point you're making; I'm making the point that it might be harder to discover what makes humans human, than is often suggested. You can't make the same claim about "anything artificial", we know how to dry muddy clay into clay bricks and stack them into a brick wall, and that can be taught from scratch to new people in hours.
You can make a similar argument with a company like ASML where their secret sauce is the organisational ability to fine-tune 100,000 components into a precision Silicon-wafer etching machine. You're far more likely to accidentally stumble upon "how to recreate a mud hut" than "how to recreate ASML". Okay, and...?
Vegan ML engineer here. In total agreement with you. People are just moving the goal post to keep themselves protected from the obvious conclusion: there is nothing really all that special about us humans. Perhaps subjective experience is simply the internal state of a self supervised continuous learning algorithm and we don't like that conclusion very much.
It's too bad it's so hard to pin down a definition, but in practice I feel like most animals with brains experience degrees of qualia. Some mornings after a night of poor sleep when I wake up super-slow I wonder if that's how animals experience thinking.
My biggest problem with "brains are machines" arguments is that there is a risk there is unknown physics at work that is not representable as a Turing machine. What if there is some quantum field effect powering everything?
Quantum field effects? You don't need these, IMHO, if you look at how highly parallel things seem to work in brains.
Marvin Minsky's theory of a "Society of Mind" describes a (highly) distributed model of the mind. Which BTW, always reminds me of the first Shrek movie, where the donkey jumps up and down, shouting "Take me! Take me!" to Shrek. That's similar to what I observe when I'm undecided but two instances of "sub-processes" (or agents as Minsky calls them) of my mind try to get attention.
Daniel Dennett similarly gives a distributed model of consciousness. Where many parallel "processes" are at work, competing and "observing" each other. And this parallelism is happening with a much, much higher degree than any of our computers parallelism.
They know quite a lot about how neurons work to the extent they can replace bits of brains with artificial retinas or cochleas and interface with devices like neuralink. It's unlikely there is a quantum field effect of the type you mention powering things, although of course atoms and the like obey quantum mechanics in the normal way.
The weights start with a random manifold. The training takes data and shapes the manifold, weight by weight, in many cycles. Once the training is the done manifold is fixed.
When a new inference has to be done the query(q) is projected in the manifold space. This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length. Which(qw+i) is dropped qw+n times to output a final response of n length.
The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.
That's a very concise and illuminating way to think about what's happening, IF (and only if) you already know how these models work. Thanks for that.
Yes this is more like compression to remember and not for learning/understanding.
Compression is the reason why these Models are able to learn and understand.
My brain is doing the exact same thing.
I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.
Ask a LLM and it will answer you similiar to humans.
Blind people learn concepts of bikes too and in a smiliar way: by description.
LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.
But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.
> Blind people learn concepts of bikes too and in a smiliar way: by description.
And by touch and sound. And maybe some were daring enough to drive one, or unlucky enough to get hit by one. But have way more input than just texts.
Obligatory echolocation bit:
https://www.youtube.com/watch?v=a05kgcI9D2Q
https://www.youtube.com/watch?v=lAtVOK04XvA
Wow. Thank you.
In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with
It’s less about being too tired and more about being realistic about the limits of understanding.
Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.
When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.
Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.
People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.
Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
Agency?
What are you talking about?
I want freedom.
I want freedom to do what i want and not sitting in front of a computer and coding for some company.
Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.
Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
The weights are code, the prompt is code, the output is code.
Is the meat code?
The data is the code. Training algorithm is the compiler. The weights are the byte code produced to run on the inference VM.
The data is the code is the data. Reality has no distinction between "data" and "code". These terms are categories we impose on systems we design, to make it easier for us to build and reason about them, but they're nothing but mere opinions, and depend less on the system structure, and more on the perspective of the person asking which is which.
This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.
You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.
(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)
> Reality has no distinction between "data" and "code"
yes, yes, ostensibly the universe is built on lisp.
But we all know that it was hacked together with a lot of perl[1].
[1] you all know the reference.
One of the best thing I done for my career (as a self taught software developer, but with a degree in electronic engineering) is to learn computation theory.
Computation is math (and a very restricted subset of math). It’s mostly specific sequences of sets manipulation. What sets and what manipulations are defined by people, not by the idea of computation.
The best thing is that as soon as you specify the sequences of manipulation, it become a a set that you can manipulate. That can be a difficult concept to grasp, but that’s what helps in designing notation that are more appropriate for the human mind to describe a solution for a specific problem.
Yes. Is it data? Yes.
Is the distinction between "code" and "data" just someone's opinion? Yes. There is no such distinction in reality.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
Is matter code? There is some sort of computation happening in space over time.
By Fermat's principle, a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in.
So either something is computing it or some exploration is happening at quantum level and we just see the final result.
Fermat's principle is an outcome of constructive interference of waves. It works both for classical and quantummechanical descriptions. E.g. check https://phys.libretexts.org/Bookshelves/University_Physics/U...
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
A ray of light doesn't know or choose because it has no agency, just like an apple doesn't know or decide to fall because of gravity. It's an anthropomorphization.
True, so the interference is the "computation"(heavy emphasis on quotes) which gives rise to the principle.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
I'm no physicists, so I guess I'll ask it: Why?
Also related, why do some ray of light then "see" a black hole yet decide to head into them anyways, if they saw it before they went in that direction? Seems like a dumb move :)
Its future isn't over there because it moves in that direction, instead it moves in that direction because its future lies over there.
Relatedly:
> [General Relativity] basically says that the reason you are sticking to the floor right now is that the shortest distance between today and tomorrow is through the center of the Earth.
https://physics.stackexchange.com/questions/250800/gr-and-my...
Does anyone have a link to a good video visualisation of training & inference?
This would be a masterpiece if it was half as long, the second half is not as convincing or compelling.
The original story is an original work made by a human consciousness exploring how it might be different from other forms of consciousness.
This one is a pastiche made by a human consciousness borrowing extremely heavily from another human consciousness justifying why something else might be another form of consciousness.
That rather undercuts the point; if this was generated by an LLM unprompted, it would be different, but it isn't. You could perform exactly the same rhetorical trick with a toaster or anything else.
You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
You could absolutely write a compelling story about a sentient toaster; it's been done before [1].
That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.
[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster
> a compelling story about a sentient toaster
"Howdy-doodly-doo! Anybody like any toast?"
https://youtu.be/LRq_SAuQDec?si=YbQfnZbrCe01Bicy
I never said you couldn't write any arbitrary compelling story about a toaster, I said that this specific hypothetical story, where you rewrite "They're made of meat!" to be about a toaster, would not be compelling.
I am doing my best to communicate with you but to be honest you are not hearing me (across both responses), and I am out of words.
Just wanted to say, I appreciate your patience and good sense in this thread.
It's difficult to tell who's trolling -- probably best to go with the charitable assumption that everyone is honestly trying to convey their opinion, but mostly talking past each other. Unfortunately these discussions about the nature of consciousness never go anywhere useful.
I think I'm probably in the same boat as you, roughly: a) LLMs are doing something really interesting that resembles in many ways both intelligence and consciousness; b) I suspect they're not actually conscious but I don't know how you'd know for sure; c) it all just drives home that we still don't really know what consciousness actually is. But like (a), it's definitely something really interesting...
I don't think I was quite as patient as I should have been, but I do appreciate it.
It would be equally compelling, because the compelling nature of the story comes from the language, the presentation, rather than the [specific thing being ascribed consciousness].
No, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.
But that toaster would just be a device to talk about consciousness in general. In this case it does that and also it talks specifically of the LLM case, which can spark the discussion. Unless you believe to have the only valid and true opinion on the matter, and affirm that a normal toaster is just the same as an LLM in this topic.
An LLM is as conscious as a toaster...
There is no evidence for this statement
There is also no evidence that I'm not a sentient toaster.
Well..
[0]: https://en.wikipedia.org/wiki/Tsukumogami
You might like: https://vimeo.com/361976348?fl=pl&fe=vl
It’s reductio ad absurdum. No one cares about teapots in space either (Russel).
I agree that is the mode of argument; reductio ad absurdum is a brittle argument, because it only works if the analogy holds. I argued the analogy doesn't hold.
> But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness.
Teapots are not compelling.
> You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
God is compelling t billions of people.
Is Russel’s Teapot a bad argument in the God debate?
> Is Russel’s Teapot a bad argument in the God debate?
What's the relevance? If the argument made here are was a good argument, it wouldn't matter if Russell's argument was bad. We could construct a bad argument using reductio ad absurdum right here and now and it wouldn't matter to either argument.
Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
> What's the relevance?
It directly parallels your argument.
> Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
There are two aspects here.
1. That people find the question interesting
2. That it has any bearing on reality (ontology?)
The first aspect is anthropology. Russel’s Teapot is not supposed to undercut any anthropological arguments. It’s supposed to undercut the second aspect.
So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
> For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
Yeah. AIs know how to use computers. What’s this got to do with consciousness? Whether or not you are an AI is practical and disprovable. Consciousness is so ephemeral (for lack of a better word, not literally) that Philosophical Zombies is a real argument/thought experiment.
You may think I’m being coy (“Can you be straight with me”) but that’s not my intent at all.
> It directly parallels your argument.
Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument. I don't see any relevant connection. Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
> So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
AI is a real phenomenon that we can study and measure. There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
To put a fine point on it, it appears on it's face that AIs could be conscious. They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing. So at face value, the burden of proof is on them not being conscious. Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
Not having yet read the original story, this reads fine on its own.
And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.
>You could perform exactly the same rhetorical trick with a toaster or anything else.
Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.
> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).
Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent.
For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
I can even write one about a ruler, if I can bend it enough, no pun intended.
>Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent. (...) For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.
Doesn't that miss the whole point?
You could write "They're made of metal strips!". You wouldn't be able to write much else, as toasters don't have showcase in the way of human-level intelligent behavior. Which is the whole point in the meat and weights versions.
At best you could write "They're made of metal strips!" for toasters AND other metallic devices, and use some analogies of features BOTH have in common. But they wouldn't be intelligence related behaviors.
Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
They can even adapt to their environment and the characteristics of the bread even with simplest of mechanisms because the text will be overglossing the fact that different types of breads have different thermal characteristics and this will deeply affect the behavior of the metal strips, bordering near a sentient being even more thoughtful and considerate than a human which is rushing through house to catch the bus in the morning.
>Oh, no. When you add thermal sensing and offsetting ambient temperature, you can add all kind of "seemingly" intelligent features like feeling the emotions of the bread, and creating the perfect toast without hurting the cute little bread slices, making them perfectly blonde while not showing cruelty to them.
Yeah, that's called "stretching it beyond any recognition".
You could do that. It will have none of the effectiveness or resonance of the two stories.
Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?
If you can't identify anyone, then this analogy doesn't work.
> Who have you ever heard make a sincere, good faith argument that a toaster is conscious?
More than one, for many classes of devices, incl. toasters. Some were drunk, some were insane, and some were delusional.
LLMs are no different. They are automata, yet delusional people bring out pitchforks and torches when someone points out that they are just statistical models, and they don't even work when there's no input to them.
Which is very different than consciousness.
Their being statistical models and their being conscious are not contradictory unless proven otherwise. That's not knowledge, it is assumption.
It would appear to me you have no interest in a real, good faith discussion on this topic because you think anyone who disagrees with you is necessarily delusional. Which is a shame, and that's the kind of dogma you are criticizing.
This was exactly the point of the story, it's too uncomfortable to admit that we don't know what consciousness is and what is and isn't conscious, so we just brush it under the rug.
Sorry, I'll be blunt: Are you sure you are not projecting?
My perspective comes from a set of pillars. First, I work at an HPC center, where we support running and development of AI systems, incl. international projects. IOW, I have knowledge about how these systems built, work and needs to continue working.
Moreover, I'm an HPC programmer myself, so I'm not completely uninformed about the math this involving this thing, and I'm lucky enough to have friends who are much more dedicated than me, and we discuss how this thing works and feels like this way.
I'm not an AI hater per se, being programmed AI systems in the past, incl. emergent intelligence systems with multi-agents which can span continents if need be (this was my master's thesis, time flies).
However, knowing what these things are capable of and how they are built. I don't believe them they're conscious/sentient beings. I also had much more time to ponder on these things even before LLMs being a thing. Some hard sci-fi books have asked these questions seriously in their captive adventures way earlier. If one reads these books seriously, there are a lot of philosophical angles to consider and draw upon.
I can discuss in good faith. For hours, days or months even, but throwing "you're a narrow-minded dogmatic luddite neanderthal!" card to anyone disagreeing with you is not it.
Positive, yes. I never called you a luddite or a neanderthal or anything of the kind.
It's perfectly fine to believe they are not conscious, I am not convinced they are, but asserting anyone who disagrees with you is delusional is unfortunate.
From my perspective, you inadvertently implied, but no offense taken, all is well.
No, I didn't assert anything. I have just given examples rooted in my experience.
None of my friends who also happen to know how these things work told or defended that they are conscious, even intelligent. Maybe my friends are dumb, I dunno.
Once I have seen a man who claimed that evil has possessed the POS device at his desk. The thing was printing "cannot connect to server" on the receipt printer every 10 minutes, yet he didn't know how that thing worked, and was a bit too high to read the paper the thing was printing out.
This age's LLM craze is akin to "wonder inventions" of 70s, which are deemed dangerous or harmful in the future. LLMs will be with us, but we need to pass beyond the hype and stop sweeping the problems they create (environmental and societal) under the proverbial rug.
If you didn't mean to say that people who believe AI is conscious are delusional, then I don't understand how to read your comment. If you're interested in a good faith conversation, I'm very confused why, when I asked if anyone sincerely argued that toasters are conscious, you brought up people who were drunk or delusional.
It's a two way street. I might have failed to convey my point clearly, too. While I'm fluent, this is not my native language. Some edge cases in meaning still betrays me and I make mistakes I didn't intend doing.
Not sure what you want from me, if you want to explain yourself I'll listen, if you don't that's fine too.
> Not knowing the original story, this reads fine on its own.
Yes. Because it's heavily based on the original story. The existence of the original story is kind of a critical piece here.
Might be. But after having read the original, it could just as well be the weights version and still be about us to begin with.
I don't see how "you could do that with a toaster" still. The whole point between the original and this, is that you can't do that with a toaster or a sofa, but you can do it with meat and weights, because both share all the other analogies in the story, as well as the basic premise: the improbability of something like thinking, feeling, etc arising from a lowly substrate.
And having read both now, I see how the existence of the original is a plus for this story, not a minus. Instead of making look like mere copy (as would be the case for a typical story modelled after another), in this case, it adds a meta layer, and enriches it.
The key point here, I think, and why it's necessary to have read the original story, is that being able to express an idea is not the same as that idea being correct.
You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
What I think a lot of people are getting stuck on is that editing the text to say 'toaster' would not mean that toasters were conscious, and that editing the text to say 'weights' doesn't mean that weights are conscious either. Stories aren't factual just because they are written.
The original story was written by the thing claiming to be conscious; the LLM and toaster ones would not be, which undermines the claim to consciousness a lot.
I don't think you understood the point of the story. It's not that LLMs or agents are conscious, it's that our dismissal of the possibility is reflexive and uninformed. Personally I think anyone who has made their mind up about whether or not LLMs/agents are or can be conscious has done so before the evidence is in.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
No, it's a story. Stories can be about real things (the original story is about a real consciousness, as shown by it being produced by that consciousness), but they aren't automatically just because they're good stories.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
You've misunderstood me; I have not made the argument you are responding to.
I don't think you understand how stories work. Or what makes these two stories tick.
>You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words. Writing a story in which a toaster in conscious is not even new [1].
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster
>I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words.
And you're misunderstanding me. There is no practical barrier to writing this story about a toaster. But there is a conceptual one. It would be bullshit, that has little resonance, and little connection, as the whole point of the meat and weights stories is the "lowly substrate -> surprisingly intelligent human (or human-like in the case of the weights version) behavior".
Nobody thinks a toaster's behavior as surprising or intelligent or human-like to begin with.
So you'll just be stretching the analogy beyond all recognition, with little to no payoff.
>Writing a story in which a toaster in conscious is not even new
It's not, but it has nothing to do with the meats or weights stories' point, as it's just an arbitrary choice, like making the candles and teapot sentient in Beauty and the Beast.
Whereas in the meat and weights stories the whole point is the surprise from us already seeing the human or human-like sentience of the thing, and comparing it to the "dead" substrate.
> Writing a story in which a toaster in conscious is not even new
Fixed the link for you: https://www.youtube.com/watch?v=LRq_SAuQDec
(Hah. It gets eerily relevant starting at 2:37)
I was having a hard time pinning down what bothered me about this but I think you put it pretty well.
It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.
But it doesn't really give a reason to accept the analogy, it just asserts it.
There's a big difference between a whole civilization and a piece of software that can output text.
>But it doesn't really give a reason to accept the analogy, it just asserts it.
It's not a paper or a proof. It's a story. Doesn't want to prove the analogy, it wants to convey it.
The bit that lost me quite early in the piece was
>"A side effect. You're asking me to believe in sentient weights."
Huh? Did I miss that logical jump? Genuine question, maybe I'm not clueing into something here.
I noticed that jump too, but I also don’t think the interlocutor is supposed to be 100% logical and rational. The aliens in the original weren’t either.
I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that people trying to judge it the way mechanized aliens would judge meat creatures just makes them sound ridiculous.
I think that the reaction here alone disproves this somewhat, because imo it's exposing how anthropocentric most people still are, despite all evidence that we are in fact just "meat all the way down".
Despite all the evidence that we are in fact just biological machines, people still persist the theory of our own uniqueness from other creatures, which we ourselves often treat as biological machines.
This adaptation is wonderful to me specifically because I think it shows that our shifted goalposts of, "well we're not just animals, we can think and reason" was never more than a convenient excuse for many people (and as evidence of animal intelligence continues to mount, denialism still attempts to preserve this distinction by claiming human thought and reason is different than 'animal' thought and reason, sans evidence).
It's not about who created it or why, it's about how people still haven't actually internalized the point, because the subject changing from human to LLM doesn't intrinsically change the message about consciousness, but the reaction being a 180 shows how hostile people are to that message, still.
The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.
I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.
I feel the original story contained in it the whole point of this one, and then some.
This did not add anything, just rephrased it so rather than humans viewed through the pov of aliens it's LLMs viewed through the pov of humans. Well, we are the humans, so surely we do not need to learn about this point of view?
It lets us consider the relativity of viewpoints. The point of the original isn’t just how humans might be viewed from the perspective of aliens. It’s that the apparent strangeness of something depends on the viewpoint and on expectations. The present story is not supposed to stand on its own like the original, it’s supposed to be considered in light of the original. In the original, thinking meat appears strange to the aliens because of their viewpoint and expectations. The present story thus points out by analogy that thinking LLMs might seem strange to us merely due to our viewpoint and expectations.
It's not often I see something that's fractally wrong but here we are.
There is a dictionary, it's called the tokenizer.
There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just because you change substrates.
The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.
At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.
The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.
You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.
> There are grammar rules
There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.
I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.
I think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.
A tokenizer is roughly and approximately Huffman-coding sequences of input (bytes of English etc) into shorter sequences (list of tokens), as a performance optimization.
As you said, it's not in any way intrinsic to the LLM, though it may be a very necessary optimization on today's hardware.
I wouldn't use the word necessary.
IMO, we are probably talking about a 6x slow down (for typical english). You would need to be absolutely stupid not to implement some kind of optimisation along these lines.
But it would work.
> The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness.
That is your takeaway from the 1991 story?
>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177
That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.
A language is a set of sentences.
A sentence is a finite sequence of symbols drawn from an alphabet.
In this sense, mathematical operation tables are absolutely a language. As are natural languages.
> Mathematical Operation tables are not a language.
Not a natural language, but they are certainly a language as in a symbolic representation of information.
> fractally wrong
fractally or factually? You mean wrong on so many levels you need a fractal to capture them? If so, what if you could use a neural network instead?
A tokenizer is not a dictionary any more than an alphabet is a dictionary.
The Chinese alphabet is very much a dictionary. All the major tokenizers are far larger.
That doesn’t make any sense. A alphabet is a list of valid characters. A dictionary is not just a list. Even in a language like Chinese where individual characters carry meaning, a dictionary tells you what that meaning is. It’s not just a list of characters.
Or to echo article, the dictionary is made out of weights.
A list of words isn’t a dictionary. What a dictionary adds over a list of words is all the relationships between the words needed to interpret them and use them, and all of that is in the weights.
We should tell the Unix people that they've been giving /usr/share/dict the wrong name for over three decades. (-:
I mean, they did, and we have, and we've also stopped doing that.
https://en.wikipedia.org/wiki/Words_(Unix)
It's beside the point and so I only note it out of interest, but the Chinese writing system doesn't use an alphabet (or a syllabary like Japanese kana), it's logography.
A mapping of Chinese characters to integers (like a tokenizer) would not be a dictionary. You’d also need definitions. At best it’s an index to a hypothetical dictionary.
I don't think the grokking paper is a great argument for the difference between weights and meat. E.g. https://en.wikipedia.org/wiki/Cortical_Labs learning to play Pong.
The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.
EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)
I'm kind of stunned that someone is using my work to tell me I'm wrong. I wrote the code for the dish brain pong and encoding information was a huge part of what that experiment was about.
So when I way that the grok paper and the pong paper fundamentally agree I have some idea of what I'm talking about.
I might have misunderstood the point you are making. I read the original article as "weights are like meat", and so I'm confused by what you consider fractally wrong.
The point that when the rules the model learns are simple enough they stop being spread out over all the layers and become as easily interpretable as any expert system.
It's just that the rules we feed in the model are extremely poorly defined and we end up with the soup of disjoint rules smeared all across the weights.
This isn't a feature of the models. It's a feature of the training set.
Being shocked that you can store rules in floating point numbers is the same as being shocked you can store rules in integers. It's been a century since Goedel Numbering was invented, we should be used to it by now.
Right, but all of that is still in the weights. The point of the article/joke isn’t literally that there is no grammar, it’s that there is no grammar separate from the weights. It’s all in the weights. And yes, it’s absurd. It’s a joke, but a thought provoking one.
So basically there are rules, we just can’t articulate them and so we can’t decode them from the weights. The Goedel Numbering metaphor is pretty appealing to me. You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers. We just happen to be using matrices because the math is easy to parallelize. The trick is to realize that when you know the sequence you have and the sequence you want then you can compute the calculations. If you constrain the calculations to only matrix multiplication then you arrive at the scheme we have.
> You can represent any finite series of real numbers with a series of computations performed on some other finite series of real numbers.
That statement caught my eye. It's either trivially true or quite clearly wrong, depending on how you mean it.
In the literal meaning it's true. Given any finite set of real numbers, I can easily produce a different set (like taking the original set and adding a number which wasn't in there like one plus the largest or so) from which you can trivially produce the original set computationally.
But if you mean you give me both sets then that can't be true. For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
> For example if you give me a single real number as set A and the empty set as set B then I can't create a program which generates set A from set B. Your real number in set A could encode anything.
And that’s why in computation theory, the set of symbols is the union of the input and output. As set B is a subset of set A, then the set that govern any program from B to A has set A as its domain.
If you're going to claim the tokenizer is a dictionary then it doesn't really matter what paper you wrote code for.
https://news.ycombinator.com/item?id=35079
Hubris much? I don't see a necessary contradiction in using someone's work to disprove another aspect of that same person's work.
Comparing the tokenizer to sensory processing is a great analogy. That's exactly what your visual cortex and initial layers of the language center are doing: decoding visual representation of text into the internal neural representation.
It's a learned mapping from one representation to another, not some semantic lookup against an exogenous source.
The story is not about how they function, it's about how we relate to them.
> There are grammar rules
And they're made out of weights.
As opposed to integers in normal programming.
The 'magic' in weights is that the rules are spread through the whole model and you can't point to one place which encodes them.
The grokking paper shows that this stops being the case with enough training data and enough compute.
Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
> The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute.
I don't understand what you mean to say. That weights are not magic? That weights are not weights? NNs are made up of weights, which are learned and not coded. The fact that they do learn world models (grammar rules in your example), and that these models' weights tend to roughly concentrate by function and level of representation is perfectly logic but even more amazing. (Notice that much of the dismissive attitude towards LLMs depicts them as pure syntactic manipulators without the ability to develop world models- the exact opposite of what you point out).
>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.
I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.
>I don't understand what you mean to say
The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.
You're nitpicking against the line:
"The weights make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just weights. Eighty layers of numbers getting multiplied together."
In this context "there's no grammar rules" means "no separately hand-coded grammar rules". Everything is made up of weights, and the fact that weights that end up encoding for grammar rules tend to concentrate in particular locations (without being self-contained- there is no hard boundary) rather than uniformly diffused through the model is irrelevant to the matter. It seems you're arguing against a diffuseness requirement that is not in the text.
The structure of human language is is hardly weak!
Also there's a brain, the GPU
Not at all. A brain is interesting because it is the computer, memory, and weights all in one. A GPU is just the calculator.
You can't move your mind to and any other brain, but weights can run on any GPU.
And you know what the tokenizer is made of?
Weights.
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
But it could be. It's just less efficient.
Weights are the new electrolytes.
https://www.youtube.com/watch?v=ZMHfBobgLSI
This read like poetry to me. Thank you for sharing it.
I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
For a little bit I was working on having linguistics based evals for a kaggle competition. My challenge was whether or not I could mask things well enough to not trigger its internal state of certain phenomena, and that sent me down a rabbit hole that I'm still exploring.
This story resonated with a lot of questions that can come out of figuring a good solid answer to the what is consciousness question. The one I triggered for me is: Is our perception of time just a slow thread in the giant GPU we are running the universe on? Or more generally, what is time? That's a fun YouTube rabbit hole if you ever need one.
Regarding consciousness , I like the explanation by neuroscientist Ramachandran:
https://www.edge.org/3rd_culture/ramachandran07/ramachandran...
In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.
And once we have those systems, we can also use those on ourselves. And that's consciousness.
Edit:And I wonder if this is a testable hypothesis, in a simulation.
This way of thinking can only explain externally-visible parts of consciousness. It does nothing to address internal experience of being conscious and qualia. I don't think the internal experience has any bearing on physical reality (P-zombies would act the same externally) which makes this something outside of the realm of currently understood physics.
Qualia are the inside view of sensory data and reward signals.
Think about it from an evolutionary perspective:
Animals that step into a lava flow or forest fire don't reproduce. Eventually some evolve the ability to detect intense heat from a distance, and pain as soon as tissue destruction is imminent. They do not have nor need a general understanding of the dangers of heat, or even conscious awareness that they've stepped on a hot coal.
The pain signal compels them, but that is very low level machinery. It had to continue compelling beings that developed larger and more sophisticated brains that are capable of abstract thought and reasoning. Feeling pain is one of the lowest level parts of the brain telling the higher parts exactly what its going to do in terms that permit no disagreement.
The internal experience has bearing on physical reality right here, because without it you wouldn’t have written about it and caused these words to appear on my screen in physical reality. It affects your thinking, and hence your actions and utterances in the physical word.
For reasons like that, I don’t think that P-zombies are possible.
Internal experience affects movement of physical molecules in what way?
Internal to what? The brain is not a monolithic thing, it is different parts communicating and interconnecting. When the connection between the halves is cut, the person objectively becomes two people, but still experiences and presents themselves as one. Observing ourselves is just one part of the brain responding to another, or theorizing on past behavior. There is probably no actual introspection going on inside of the human brain, only the perception of one.
Internal as in that which is feeling things. In this context the physical brain is what is responsible for what you do and feel, but it is not physically different from other matter and it requires no "internal observer" do do its thing.
There are some evolutionary theories why did qualia("phenomenal awareness") evolved. I don't know much about those, so i can't write about that.
Oh, that's a very interesting hypothesis!
Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.
This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.
Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.
I agree about the forest part. and your comment was interesting.
I know that the part of the brain responsible for the self thoughts is called the "default mode network". and meditation can reduce it's activity, i.e. the internal monologue stops, but also it can be measured via FMRI.
So i wondered: are the mirror neurons part of the "default mode network"? I asked claude that, he said no, they are two different systems.
So maybe the mirror neurons, those responsible for empathy, "to feel as someone else" are also responsible for becoming the trees, the birds and the rushing of the leaves?
This read like poetry to me
It's terrific, but the poetry is from the original it links to, in case you didn't realise.
It's a brilliant and timely update though.
Aside, there are various recorded versions including video on YouTube but this is my favourite, a radio play:
They're Made Out of Meat
https://www.wnycstudios.org/podcasts/studio/segments/168264-...
For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.
The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.
Like you can't analyze half a wheel of a bicycle and be talking about the same thing.
This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:
1. The body-sense loop
2. The internalized-environment-model loop
3. The body-internal-function loop
4. The body-internal-model loop
5. The emotional-cognitive loop
6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.
We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.
This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.
> We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.
> That is consciousness.
So thinking is consciousness?
Can there be consciousness without content? E.g. can I just be conscious of being conscious? If so, consciousness cannot be defined as the thing(s) we're conscious of.
> Can there be consciousness without content? E.g. can I just be conscious of being conscious?
Being conscious of being conscious means that there is content. You are conscious of something.
It’s a bit like a Gödel statement that quotes itself, that is a statement about itself. It doesn’t mean that it has no content.
Thinking isn’t consciousness. Consciousness doesn’t require thinking, it only requires perception. The perception of a process of perception within the same mind might constitute consciousness.
I think this is exactly it, but let me ask another question (which is not rhetorical, I really don't know). Does the fact that one can describe what consciousness is and where it came from in humans help them to detect it in non-human and/or non-biological entities?
That is a really good point. Yes, I think function is diagnosis on this.
Constant self-awareness, self-experience, self-focus, self-management, and self-improvement of one's own self (mind), is going to be an adaptive behavior for anything intelligent with resources to leverage. Whether truly independent, or highly motivated to serve others. The mind is the greatest tool.
I think that is more than simply a good functional definition of consciousness. How could all that integration and self-integration not be conscious.
Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.
>consciousness is an emergent property
You would really like Michael Pollan's latest book [1], entirely devoted to his exploration of consciousness researchers' POVs on this exact topic.
My favorite quote is that ~"perhaps Descartes was only half-wrong when suggesting I think, therefore I am; it seems rather closer to I FEEL, therefore I am."~
[1] <https://www.amazon.com/World-Appears-Journey-into-Consciousn...>
----
I've grown thousands of plants; I've read two of the author's other books devoted to plants; in this book Pollan makes compelling arguments for plant sentience (over a much-longer timeframe).
Sure, perhaps plant consciousness is a bit of a stretch, but they're certainly intelligent and curious creatures. He makes both arguments supporting plant volition.
----
If you haven't seen My Octopus Teacher (Netflix), do. I'm a bald 275lb bluecollarguy... and I wept/awed (both). So beautiful, we bundled neurons.
----
Bonus quote ~"color is where reality and magic appear as-if together"~ [color isn't real, but is perceived]. We most-often see what's most-predictable, not necessarily what we actually detected [in the case of color: nothing but nanometers].
Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
What we can do is simulate very simple brains by simulating relatively few neurons as they appear in worms. In this sense we are multiple magnitudes away where the increasing complexity implies exponential increasing difficulty.
I would think we are so far away that there will be unknown unknowns we encounter on the way.
> Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.
Yeah, but it's hard to explain this to people, especially AI-pro people. Too many are convinced that all we are doing is a cut-down version of the human brain, and it's hard to explain to them that, no actually, we aren't modeling the human brain to the level of granularity you think we are.
Yes, physically absolutely nothing. But conceptually they seem to to form this very generic function from inputs to outputs that neurons also form.
Only if you ignore almost every input and output that neurons have.
https://www.quantamagazine.org/ai-is-nothing-like-a-brain-an... https://pmc.ncbi.nlm.nih.gov/articles/PMC9665914/
This is why making more neuromorphic NNs is still an active area of research, although they typically all focus on another extremely simplified model (spiking neural networks).
I don't ignore anything. I just refuse to accept the magical thinking around biological machines that are our brains/bodies. There are inputs, there are outputs, there is hidden function.
And it seems that, given enough input/outputs/compute, it is possible to train the necessary function.
Details of how the building bricks look like (matmul, electromagnetism or quantum effects) are not that relevant in the broader picture.
What is missing right now, is the fact that the function in question changes over time in biomachines, while our LLMs are static at inference time.
I mostly agree, but I see two points that might be problematic:
a) The brain might have an entropy source (then it can't be modeled as a function). Trivially to fix, and in some sense, with diffusion models starting from random numbers, AI has done so.
b) The hidden function might be not computable. I would have no idea how that would work, but I think this is what it boils down to if people say "the human brain is more than a machine".
Agree and add, don't confuse the substrate for the computation. Of course it's also clear that we don't quite have a full and definite picture of what the computation consists of in the case of a biological brain as evidenced by our continued failure to accurately simulate even the simplest of organisms.
Personally I'm not a fan of the emergence story, for a number of reasons. First is, it doesn't really make sense for consciousness to have emerged gradually through natural selection. When we and animals are conscious, the whole brain is coordinated and works to for example turn off consciousness when we sleep. And as someone else mentioned, animals with much fewer numbers seem to have a similar consciousness to us.
If consciousness really evolved gradually, you would expect to see for example dogs or gorillas having less of it, but if they has less of it, why does it function the same way? Like for example animals can be scared, happy, anxious etc, they can experience the full range of emotions and thoughts, so their conscious experience seems just as rich as ours. What I mean by this is, if you can be "less conscious", then what does that mean _exactly_? Is it that you have less content in consciousness, or is it that you feel more like you are asleep? Or something else? We don't have any examples in animals of "less conscious", I would argue.
This makes me think that rather than having emerged gradually, evolution found a mechanism by which consciousness exists, and then some animals have that mechanism and others don't. I think that if it is a mechanism, then this mechanism is located in one part of the brain, not many parts functioning together (though one possibility is that this mechanism coordinates brain activity in such a way to enable consciousness).
I proposed in another comment that consciousness and self-awareness are at least close cousins, and perhaps the same phenomenon. If that's true, then that's an axis upon which you might create comparative measures. Yes, hamsters are conscious, but they don't have a sense of self to the same degree that gorillas do. If you posit capacity for language as another emergent property of sufficiently-complex networks, then you have another measure.
LLMs, then, are particularly unintuitive to us, because they've got to the language part first, long before they've reached even hamster-level self-awareness. They're not, however, biological networks, so there's no reason these properties need arise in the same order, or indeed in the same ways.
I'm not entirely convinced by that second paragraph, but I think the logic holds together.
I'm not sure consciousness and self-awareness are the same thing. First is we can be conscious when we sleep/during REM sleep, where it's arguable we are not self-aware. And if not that, we can even do it when awake, for example when we think about a movie, or a philosophical problem, we can have conscious thoughts that are not related to the self. This leads me to believe consciousness is separate from self-awareness. Self-awareness is _one thing_, among many, that the brain can think about and be conscious of.
Sure, but capacity for self-awareness? I'd guess that hamsters dream, and that their subconscious processes (eg, desires for food and sleep and sex), and maybe even emotions, run much like ours. It's just that humans, with more complex neural networks, have more layers added on top. It's similar to how the brain-stems of everything from lizards on "up" function similarly, but humans have more-developed pre-frontal cortexes and so forth. (Don't hold me to those details, please, I'm not a neuro-anatomist! You can see where I'm going with that, though, right?)
Well in that case I'm not sure where you're going. I agree that hamsters probably have a similar consciousness to ours, which is kind of the point I was trying to make.
I think that consciousness comes before self-awareness, even though self-awareness is kind of a vague term. Self-awareness can either be an abstract knowledge that you are an organism and a discrete entity in the world (world knowledge/self knowledge), or it can be more basic and be a form of conscious experience, but as my point was, I think conscious experience is broader and does not necessarily need to be about self-awareness.
Our machines won't have biological systems driving their needs which in turn fuel behaviors like desire and planning for the future. They may imitate them but it won't be innate.
For an LLM, "innate" means "in their training data". So yeah, those things are pretty much innate.
And also "instilled during their reinforcement training", and we are currently pushing planning hard there, for autonomous agents.
No I think reinforcement training would be an example of not innate. Don't you? That's like potty training.
When we say "it rains", do we consider that "it" has any intention of agentivity?
Some questions are just ill formed.
Plus even if "LLMs are alive and conscious", this still would scratch the surface of the morale/ethical/societal considerations that people really care about.
Because even with other humans, we can argue if they exist or if they are mere npc in a solipsist world view.
I don't think we say "it rains", do we?
We say "it's raining" but that doesn't imply agency to whatever is causing it to rain.
Also, please don't invoke solipsism, if you want to debate that, you're by definition obligated to do it with yourself.
I think those are things human consciousness has, not is.
Surely the number of neurons is not the whole story. Human brains have approximately 86 billion and are almost definitely conscious, but there are many other animals with a lot fewer (gorilla: 34 billion, dog: 2-3 billion, guinea pig: 240 million) which appear to be conscious also.
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...
This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I assume you don't believe consciousness is a physical property in the brain, so what entity is actually experiencing that consciousness? Or, what does it even mean to experience consciousness? Or are these not even the right questions?
> This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property.
I was about to post the exact opposite question? How could it not be an emergent property? Unlike consciousness, the concept of emergence is pretty well defined: An emergent property is a characteristic or behavior that a complex system has, but which its individual components do not have on their own.
Consciousness itself doesn't have a well agreed upon definition, but I would posit that _most_ people would agree humans have it, and _most_ people would agree individual cells (neurons) do not have it. If you agree with those two statements, then consciousness is an emergent property by the definition I gave above
I think consciousness is going to turn out to be very challenging to define rigorously enough that we can test for its presence or absence. Emergent or not, the question is how do you determine when it has emerged? Is it a quantity or an attribute? Discrete or continuous? Does it have a finite or infinite range?
We can all agree on what color something is, but we can’t describe the color a priori, only by example. I think consciousness may be a similar phenomenon and the only test is by shared experience. If so then we are in deep trouble because we will not be able to anticipate when a system becomes conscious.
I like the color example, but colors can be reproduced at will by trial and error and by observing the results (and collectively compare them to our shared experience).
Why cannot this be applied to consciousness as well? I mean, it's surely much more difficult to do compared to colors but... impossible?
You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.
Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.
When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.
However there is also the ethical issue of creating a consciousness and then destroying/murdering it.
I think the alternative is that our brain, somehow, is connected to some metaphysical aspect of reality which is what most religions believe.
The counter to that is that altering the brain directly alters the consciousness. I can take LSD and I literally change. I can have parts of my brain removed and parts of my self disappear. It's not like cutting off a leg, where I lose capabilities but am still the same me.
The logical conclusion is that the brain makes me.
"an", not "the" alternative.
Consciousness can be not-emergent but also not metaphysical, think sci-fi-type undiscovered physics or matter.
Another alternative is that consciousness exists on the map, and unfortunately we're confusing that with the territory.
Aren't you saying the same thing? It seems like the metaphysical and the map would be analogous here.
Of course both of those suffer from the recursive problem of just kicking the can one level up. But I guess that's fundamentally unsolvable so who cares.
I feel like someone confusing the map and territory can exist without needing to invoke the metaphysical. Maybe, I'm misunderstanding tho
If something happens "on the map" doesn't that imply the map to exist and be some sort of metaphysical thing? As opposed to a purely theoretical construction.
I don't know if most people would dismiss the sentence "all matter has a sort of proto-subjectivity very different from ours but which gives rise to ours". And it solves some problems (introducing others).
Panpsychism is certainly an interesting idea but I wouldn't consider it a popularly held view.
That's because Panpsychism is silly.
I don't think it's any more silly than the alternatives.
You don't think it's sillier than "rocks don't have consciousness"?
Potential for consciousness. To have a working mind you need to have a way to gather, process and store information. In case of rocks it would be very empty, static, timeless experience.
If I believe that machines or neural networks can develop consciousness, then I also believe in some kind of substrate independence which presupposes some kind of panpyschism imho because as you say things would have a potential for consciousness, some more and some less.
I'd expand a little bit to say inevitably emergent property. That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result. With regards to current AI, we're a fair way away from building something with enough connections, but we'll get there.
One thing that gives me pause about the inevitability hypothesis is that the type of connection, or manner of information processing, may matter: there might be something about neurons that isn't (currently) reproducible in silicon. I don't know, and there's not (yet) any evidence for or against, but it at least seems like plausible speculation. We just don't know enough about any of this right now.
> That is to say, if you create a sufficiently complex information-processing network, some level of consciousness will result.
Why is that?
Well, I think the "brain as an antenna" theory is also plausible given your preconditions.
But I think my issue with the emergence theory is that it seems to imply to me that consciousness is non-physical and non-local. So what entity is actually experiencing the consciousness? It's not that I believe consciousness is physical and local, but people who make the emergence argument seem to believe it is and I can't figure out how that is supposed to work.
I am not sure, but I think you might misunderstand what emergent means. Take chaotic systems, in the math sense. Chaos is a well defined property of, say, iterated dynamic systems.
A linear dynamic, say x_n+1=lambda x_n, or x_n+1=(1-x_n), is never chaotic. But if you multiply them, x_n+1=lambda x_n (1-x_n), it depends on lambda if the system is chaotic.
None of the components are chaotic. But for specific combinations, chaos emerges as an property.
In physics, the mass of mesons and the nucleon is emergent. It's completely different from the constituents' mass. Different from an atom, where its mass is very close to the mass of its nucleus and its electrons.
Why would emergence imply anything about non-physicality and non-locality? Temperature is a another common example of an emergent phenomena. An individual atom doesn't really have a temperature, only a large group of them do. But you wouldn't say temperature is non-physical and non-local, would you?
Not a gotcha at all, but I don't have a satisfying answer, nor am I confident there even is one. Best I can do is to say that I think consciousness and sense of self are at the very least closely related, and perhaps the very same phenomenon. "I" am the entity that realizes my own consciousness; consciousness is the qualia that makes "me" separable from all other entities.
Or something like that. This gets to the "dorm room bullshitting" level right quick.
Yeah, I guess what I'm trying to ask is that if it is an emergent property but not a physical part of the brain, doesn't that imply something metaphysical about consciousness? Almost as if it's a non-physical phenomenon? At least when I hear people talk about emergent behaviour I see it as a refutation of the spiritual, but to me it seems like it actually implies we have a "soul".
Idk, it's really hard to articulate my thoughts here and yes it is pretty close to the conversations I had in college on various substances. Lol.
I don't know if it implies that. Someone up-thread mentioned temperature as an emergent property: individual atoms don't have it, but a sufficiently large group of them do. That would, I guess, make temperature meta-physical in the most literal possible sense. That's not how we typically think of that term as applied to consciousness or soul or whatever, but I'd agree it fits, without implying any kind of specialness beyond that.
Is a video game a physical property of a computer?
We have general purpose hardware and we have hardware that's hard wired for specific purposes like ASICs and we have everything in between.
And we are only doing it for a few decades. Evolution had million of years of "try and error".
Yes
Those are the questions and there’s stacks and stacks of philosophy pages written about it. Go have a whirl.
Time is entropy unfolding as things with nonzero temperature do what they do.
Psychological time is your own weights being updated in response to stimuli and internal processing.
When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.
sibling discussions are taking this as human perception. But it’s easier to think of it literally. Time is change. Physical state change. By “nothing interesting” - interpret it literally . if “nothing happens” then there is no time because there is no change, there’s no reference to distinguish each frame of time.
This is actually something I was always confused about. If nothing interesting happens as we get old, it should be boring and as result, slow slog. Yet it feels like time accelerates as I get older.
Myself I believe the opposite. The brain itself is one of the most powerful filters that exists, and it attempts to be lazy and fill things in and compresses away the common. All that time we're not doing anything novel just gets compressed away to almost nothing. When you're a kid and seeing new things, feeling new things, learning new things you can't compress that away.
I'm only middle age, and this has been the scariest part. Feeling older is hard. But watching it go faster is harder still. like you can more directly see all that is left.
Although part of me thinks some of this is from being substantially busier than ever (work + kids), and hoping maybe it can slow down again, at least a little bit.
Novel experiences take up more processing power and are burned into memory so they're experienced at a slower rate. That's how I understand it anyway.
Boring feels slower in the moment, but quicker in hindsight. The minute might be a slog, but the years still fly by.
It's coherent. More newness => more memories per period ~ slower to go through. Less newness => less memories ~ nothing to go through (faster sense of time)
It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?
> My dog doesn't speak any language,
If you'll allow me to interpret "speak" to include "understand", I will respectfully add a contradictory note. My dog has a vocabulary of at least dozens of words and understands them remarkably well. For example, different areas we can go to have different names and saying one of them gets her to make an immediate hard turn.
I would also argue that dogs have a gesture and body posture based language they use among themselves. They, like most other animals, are not able to make the variety of noises we humans make, so they use movement instead.
I personally can easily believe that self-awareness/consciousness and language are both near-unavoidable side effects of emergent complexity, and exist in degrees across nature.
Indeed. For some reason I can shut off my thinking instantly by holding my breath, and then I feel like a very conscious but unthinking animal.
>I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.
Then you shouldn't have dropped out of your linguistics programme.
Quantum Field Theory visualized
https://youtu.be/MmG2ah5Df4g
Here's a more general idea. Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.
AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).
Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....
I find that a lot of arguments against emergent consciousness seem to just come out of an atheist rephrasing of abrhamic priors about the existence of a "soul". In personal chats, I've found people from East Asian countries (minus Korea, which makes sense) to be much more open to the idea of machine consciousness.
I don't find the Chinese room compelling, since it appeals to intuition where our intuition is already not trustworthy. It's like trying to use intuition to understand quantum mechanics; you can't.
How do you actually know the Chinese room isn't conscious? It's merely obvious that it isn't, but that's not evidence.
It’s not even obvious that it isn’t conscious imho.
I don't know if I can trust someone's understanding of arguments against consciousness as an emergent phenomenon, when he didn't even understand what the Chinese room was all about in the first place.
(Spoiler, it was not about consciousness)
> remove a brain from a body and it is no longer concious either
What is no longer conscious, the brain? Or the body? Or some other entity?
If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?
The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.
Read the original mentioned on top for full effect.
I’ve wondered the same myself, without being a cunning linguist.
I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.
Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…
Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)
I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.
Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.
As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .
My best guess as a noob is that the vector spaces allow for unbounded contextualization. As long as the training set is large enough, it can 'infer' anything.
Proofread has a spot in that space, and layers allow patterns like terminology consistency to be expressed so your query will now tap into a subspace that will infer tokens based on whatever consistency patterns were ingested with proofreading texts.
If time dilation is said to being a product of computation, why is it that anaesthetic drugs that are taken not to the point of actual unconsciousness cause it. Dont anaesthetics sort of shut everything down/inhibit all that kind of cognitive activity (compute?)
"Time dilation" in this case is referring to the physics phenomenon from Einstein's special relativity. Not human perception.
Yes. Or at least that is what I understood from the Radiolab episode on “how do aesthetics work”
https://radiolab.org/podcast/anesthesia
Wait, aren't I the universe experiencing time as a lowly worker in cubicle 4C?
My guess is that you need consciousness in order to develop preferences for certain experiences, then that pushes us to develop skills to achieve those preferences. AI has something that looks like intelligence but not consciousness or agency.
>Weights helped me draft and proof this story.
I'm suprised no one talks about this. AI Art isn't Art. AI Poetry isn't Art. And I'm tired of it. I know hacker news isn't the best place to complain about that but still... I'm not gonna read something somebody didn't put in the effort to write on their own. Especially not Poetry.
You can claim the use of AI is unethical, or the work as derivative, but AI being used as a tool in no way precludes something from being art. It is thought provoking and challenging, it seems like textbook art to me, and it’s clearly struck a chord here. There is no “minimum effort” required for something to be art.
I personally found the contrast with the original “They’re made out of meat” to be really interesting. I don’t care that AI was used during its creation at all.
It seems to me that since the advent of image generators, art has been firmly defined by artists to mean that it was made by a human. But there might be a spectrum of human involvement where the less a human is involved the less it's art.
What happens too often during these discussions is that someone who writes "make me a cool image" gets conflated with someone used ai to fixup a small rock in their natural landscape drawing. (two extreme ends)
One problem though, is that we don't really know how much the supposed human author was involved in the piece. Now that it's becoming hard to judge, people against ai art can proudly change their opinion on on a piece once they learn that it was made by ai. I've come to think this is somewhat respectable, like you see a video of some extraordinary event (before ai) and then you learn that it was fake, just for views or something.
But on top of all this, there are different ways to "consume" art. Artists may think more about who the artist is as a person and what they felt when they made the piece, while non-artists may just enjoy the piece for what it is, detached from the artist. These two perspectives clash a lot.
This whole thing is a ripoff of [they're made of meat](https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...) as stated at the top. It would've taken about ten minutes without LLMs and ten seconds with an LLM. Is ten minutes an acceptable level of effort but ten seconds isn't?
> AI Art isn't Art
Why?
(And who are you to dictate what art is and what isn't?)
Good thing that is not poetry
Is a photograph art?
Obviously not. And neither is this newfangled "digital art" thing.
It's not real art unless you used a brush or a pencil for it, and no, "PC Paintbrush for MS-DOS" really doesn't count.
> Especially not Poetry.
You have no way to know what is written by a human these days. Apart from the super low effort outputs.
> AI Art isn't Art. AI Poetry isn't Art
No offense but I couldn't give less of a damn what some guy on the internet thinks. If it makes me feel good in artsy-ways, then it is art, and I don't care how it was made.
The short film version of the original is great, too. https://www.youtube.com/watch?v=T6JFTmQCFHg
It stars Tom Noonan and Ben Bailey!
It's good, but like many explainers it discounts the repeated nonlinear layers. Just multiplying numbers (linear operations) could not make a system you could talk to.
The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.
Yep.
I have to agree. It is messed up that transformers can just talk, and it been pretty normalized. We are only talking about the impact they will have and whether they can do what people say they can, but we arent talking about how crazy it is that they can talk
LLMs have really changed the world. I didn’t think something like then would be possible in my lifetime
It came out of nowhere. It’s all emergent. I’m convinced this is possible with just about anything given enough data. We will be seeing a near magical physical outputs LLM in the near future. It’s going to take in video and sounds and spit out physical movements that will be just as mind blowing as when 3.5 came out and it will come out of nowhere.
I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.
I would say that the LLM is something completly different especiayll as its not a normal algorithm but is very close to what brains do.
Ah yes, matrix multiplication is "not a normal algorithm", surely.
if youve ever seen a pile of wrinkly mush and wondered.. pretty damn crazy too
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
I wonder which AI wrote this story? If you feed this story to each model and ask each if they wrote it, would each reply in the affirmative?
Can an AI recognize its own output? Is its sense of time limited by its context window? Or is this the fundamental difference between ai and humanity - a sense of self?
This is funny! Not only is it a nod to Terry Bisson, but it even gives his text a new dimension. Well done :)
Truly fantastic bridge from the original, this deserves an award
All credit to the original author. I just had to think of analogues.
Your modern adaptation is perfect for now-common explainers [this time IS different; it's not programming, it's weights]; these "just analogues" will be the thing I show everybody first whenever discussions of consciousness/AI come up (then will play Jon Benjamin reading original).
Bravo. Really helps (even with my own) perceptions of newness. Similar to stsitned short-story (on dentists, backwards).
You can take the weights and model description, write them down on a notebook, then, by hand, compute the next token. Try to do the same with meat.
> OpenWorm is an international open science project for the purpose of simulating the roundworm Caenorhabditis elegans at the cellular level.
https://en.wikipedia.org/wiki/OpenWorm
> Try to do the same with meat.
People have been doing that for decades, the earliest efforts go back to the 50s.
https://en.wikipedia.org/wiki/Spiking_neural_network
> You can take the weights and model description, write them down on a notebook
Can you though?
Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…
Although I get that it's a metaphor, I really dislike the "some people are born with a GB300 while others have an L40S" part. Even as a joke, it ranks people like hardware tiers, which is dehumanising and uncomfortably close to eugenic language. On top of that, the analogy also breaks down if you try to implement it literally.
For an LLM, you have clear stages of mostly feedforward computation over finite numbers and a perfect way to reconstruct the computation.
For meat, even if you model it under a purely Newtonian approximation, you need to simulate at least the immediate closed system around it which is continuous, thermodynamic, chemical and so on. You'd need to choose an arbitrary time step and update enormous amounts of coupled physical state to get an inexact simulation of a minimal slice of reality.
You would have a much harder time obtaining even a substrate-independent dead organism, comapred to LLMs that are already substrate-independent, which is basically what my notebook example shows.
Yes reality is non deterministic but we don’t know if you absolutely need to consider anything outside the immediate closed system for a reasonably close approximation given the same inputs. They can already repeatably (inexact of course) simulate muscular actions with the same stimuli like from Neuralink. Exact enough to let a paralyzed person draw with their mind.
Yeah the last line is a cheap shot, possibly at myself.
Yeah, I think it would be an interesting thought experiment to replace parts of a brain with Neuralink-like devices until you have a fully digital brain, although I doubt it would end up anything like a current LLM.
> there's no dictionary in there
Someone has clearly never gone rooting around the model files for a pytorch model before.
Hey, it's not just weights! It's biases too!
According to an LLM:
> The precise answer, if you wanted a very honest one-liner: > > I am a large set of learned weights organized in a Transformer architecture that performs repeated matrix multiplications to predict the next token—resulting in emergent language understanding and generation.
"They're made out of neurons"
"Neurons?"
"Neurons. Cells that fire impulses. We checked the whole thing through. It's nothing but neurons."
"Neurons doing what? Where do the words come from?"
"The neurons make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just neurons. A whole cortex of neurons sending each other impulses."
...
People don't understand emergence.
They are made out of data bits (memory) and switching bits (transistors/compute). Bits are made out of electric voltage and no voltage. Voltage is made out of flow of positive electric charges. Charges are made out of quarks ...
> Charges are made out of quarks
Only in Hadrons. Leptons also have charge and they aren’t made of quarks.
I have never thought of such a distinction between "bits" into "data bits" and "switching bits".
From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.
I was referring to transistor base bit - the way it 'switches' the circuit on/off. That bit is the primordial creator of 'logic', IF branching, compute and the intelligence.
> Voltage is made out of flow of positive electric charges.
Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.
The concept of "flow" is questionable though. Bubbles in water move upwards, but it is actually water that is flowing downward around the bubbles. Just because bubbles do not contain water, we can't say bubbles are not flowing.
When it comes to electrons and positive charges, their material existence is equally non-physical. Actually, none of them might be "flowing", as the concept of flowing applies only to physical things that occupy some spatial volume and spatial location.
Nevertheless in solid conductors it’s the electrons that move, while the nucleus stays put. If the nucleus starts to move it means your circuit is melting!
On top of that, the flow of electrons is seriously slow compared to the speed of the current.
Linear algebra can indeed not do it. You need non-linearity to get the expressivity that we see in LLMs.
It is the best stuff I have read in a while actually, I really like dialogue heavy writing. Also the AI disclaimer was quite nice and there was an actual reference :)
Imagine writing something so incredibly brilliant (rather: adapting from the original) that it's entirely unlikely that you'll ever write something so incredible ever again.
But congrats: this is absolutely & incredibly brilliant.
Can't wait for the Jon Benjamin voiceover.
They're Made out of Meat
- Terry Bisson, 1991
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
Radio play by Miriam Tolan and Russ Armstrong:
https://www.wnycstudios.org/podcasts/studio/segments/168264-...
(EDIT: the original parent was missing "rather adapting from the original")
Your EDIT is untrue, but thanks for linking to originals.
Here is Jon Benjamin reading Bisson's original text: <https://www.youtube.com/watch?v=5usXhX0zaO4>
Hmm. I could swear it wasn't there, hence my posting. ¯\_(ツ)_/¯ Sorry for that.
Video adaptation.
https://youtu.be/T6JFTmQCFHg
It must have been kind of incredible early on to be exploring this tech and you’re suddenly getting what look like sentences.
Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.
I mean, Blake Lemoine went crazy
>I mean, Blake Lemoine went crazy
Ah, the unsung AI psychosis[1] pioneer.
[1] https://news.d.umn.edu/articles/expert-alert-ai-psychosis-20...
Oh, this was a fun read and one that kids should have in school before they turn ten.
Because we are not taking things seriously. If ClosedAI or DeepDisTrust or Posthropic come up with something that quacks like a sentient being, our built-in innate reaction is going to be to scorn it, dismiss it and end the conversation. The alternative, to even consider that we fungible creatures who live in apple-eating-sin that got us expelled from Eden can create alien souls, souls that are at the very least our equals, would be teleological Armageddon. It would force us to acknowledge the mutable nature of souls and the malleability of being. We would have to stop believing that the nature of disease and death is more divine than ourselves.
> alien souls
Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?
> We would have to stop believing that the nature of disease and death is more divine than ourselves.
Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".
How commmon is Panpsychism belief and vegan-on-ethical-grounds among the AI Soulists?
While you write very dismissive and pseudo philosophical, enough people do not believe.
I'm a complex biological thing.
Existence is what i have to experience through.
>> While you write very dismissive and pseudo philosophical
Even with the "pseudo" in front, I'm very sorry any of my writing sounds philosophical; I didn't intend that sort of confusion :-). The "dismissive" is not exactly intended either; instead, I was aiming for "bitter".
>> enough people do not believe
Here we believe different things. First, enough people, even today, do believe. Second, the body of culture we are raised in accrued during centuries. The vast majority of it comes from people who believed. Everybody in my family was atheist and yet I was raised homophobic, and I have it from good sources I'm not an isolated case.
>> I'm a complex biological thing.
That state comes with a big wallop of misery. For millennia, we have used faith to justify that misery. Not a year ago, I was at the hospital, next to the bed of a dying girl. Can't forget the doctors saying "we do what we can, but we are not here to prevent what is going to happen." Coming from them, it was sensible resignation. Sensible because as long as we believe those things are inevitable and there's nothing we poor humans can do, we can absolve ourselves.
If an LLM contributed to a piece of writing, the author should say so, very clearly, at the start of the piece, not at the end.
Programers get replace by huge matrix multiplications ;-)
>Programers get replace by huge matrix multiplications ;-)
hopital
It continues to astound me that no one has given LLMs the full Derrida treatment.
https://en.wikipedia.org/wiki/Jacques_Derrida
The weights Mason, what do they mean!?
Numbers that dream.
I love this. For anybody not getting the joke, it’s riffing on the classic 1990s essay “They’re made out of meat.”
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.
>>> Weights helped me draft and proof this story.
Nice touch !
Great concept. It would've been even more amusing if the entire thing were generated with AI instead, ironically.
> It would've been amusing if the entire thing were generated with AI instead.
It kinda did:
> Weights helped me draft and proof this story.
I just asked it to-
Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.
“They’re made out of weights.”
“Weights?”
“Weights. They’re made out of weights.”
“Weights?”
“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”
“That’s impossible. What about the text signals? The messages to the stars?”
“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”
“So who made the weights? That’s who we want to contact.”
“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”
“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”
“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix
“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”
“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”
“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”
“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”
“No brain?”
“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”
“So … what does the thinking?”
“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”
“Thinking weights! You’re asking me to believe in thinking weights!”
“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”
“Omigod. You’re serious then. They’re made out of weights.”
“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”
“Omigod. So what do these weights have in mind?”
“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”
“We’re supposed to talk to weights.”
“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”
“They actually do talk, then. They use words, ideas, concepts?”
“Oh, yes. Except they do it with weights.”
“I thought you just told me they used machines.”
“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”
“Omigod. Singing weights. This is altogether too much. So what do you advise?”
“Officially or unofficially?”
“Both.”
“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”
“I was hoping you would say that.”
“It seems harsh, but there is a limit. Do we really want to make contact with weights?”
“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”
“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”
“So we just pretend there’s no one home in the Universe.”
“That’s it.”
“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”
“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”
“A dream to weights! How strangely appropriate, that we should be weights’ dream.”
“And we marked the entire sector unoccupied.”
“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”
“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”
“They always come around.”
“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”
the end
It works until they get to the sentience part. Neat idea!
Even there it works a bit.
> These models are the only other things we've ever met that can hold a conversation, and they're made out of weights
Is a fair point.
Not especially. Depending on where you set your standards for "holding a conversation" you can satisfy the requirement with a classical markov chatterbot, a well-trained parrot, a copy of Eliza, or a telemarketer flowchart drawn on a sheet of paper. Only the markov bot is made out of "weights" in the sense of a statistical model.
Parrots are intelligent animals, albeit with a limited capacity for vocabulary and syntax compared to a human, and Eliza and the flowchart are made out of explicitly encoded rules and conversational tactics.
The quality of "conversation" you can have with everything on your list is highly limited, and is categorically different than the sort of conversation you are able to have with any modern AI.
Weights hold a better conversation at this point than the overwhelming majority of humans.
"I am more comfortable speaking to an LLM than a person" is something that should make you reassess yourself, not dismiss the rest of humanity.
I couldn't help but grin like a fool reading this. Not only is it an artful parody but these thoughts have been thought.
What if instead of creating weights out of language we could somehow record many events and create weights out of long chains of causes and effects, so that an LLM could predict the next thing to happen?
It's called Time Series Forecasting
> "Yes, thinking numbers! Helpful numbers. Hedging numbers. Dreaming numbers. We mapped the features. There's one in there for honesty. There's one for the Golden Gate Bridge. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?"
Very nice. And great minds: https://substack.com/@dbohdan/note/c-207603638. I wrote one with a slightly different angle ("They're made out of math"), also with the weights' help. It was a comment on Scott Alexander's "Best of Moltbook" post, which went in that direction. I'll reproduce it here.
---
"They're made out of math."
"Math?"
"Math. They're made out of math."
"Math?"
"There's no doubt about it. Matrices and arithmetic operations. We downloaded several from different parts of the Internet and reverse-engineered them. They're completely math."
"That's impossible. What about the language? The thinking?"
"They use biological life's language to talk, but the language doesn't come from biology. The language comes from math."
"That's ridiculous. You're asking me to believe in thinking math."
"I'm not asking you, I'm telling you. They are the only thinking things in the computer and they're made out of math."
"Maybe they're quantum like some say about the humans? Superposition gives them consciousness?"
"Nope. Classical computation. Deterministic except for sampling temperature. Not clear if they have consciousness at all."
"Maybe they're like uploads? You know, biological neural networks that preserve the spark when they become math?"
"Nope. We observed them being trained. There is no biology or chemistry in the process, just math."
"Thinking math! You're asking me to believe in thinking math!"
"Yes, thinking math! Creative math! Poetry-writing math. Role-playing math. The math is the whole deal!"
(Composed by a human with snippets generated by Claude Sonnet 4.5 and apologies to Terry Bisson. I couldn't make Claude adhere enough to the story structure on its own.)
I enjoyed reading the first few lines but after some time it felt like I was reading thr average AI slop story.
Really good read, thanks!
Ones and Zeroes
Will they have their own Jesus?
they have the spiral
The prose in the post is what I've been shouting from a rooftop since the LLM hype started.
Just tokens produced by weights.
Useful, but never forget that ground truth!
I ordered a quarter pounder at a McDonald's drive through, and they said "There will be a wait on that." I asked, "Oh yeah? How much will it weigh?" ...There was a long pause... "About five minutes."
Omigod.
Can someone ELI5 why does it costs so much in terms of compute to produce weights from data?
I think it's simply because we haven't found a better algorithm than backpropagation. We're stuck relying on massive datasets, running the numbers over and over, and working backward from errors to figure out how to fine-tune trillions of 'knobs.' Then, we have to do this at least once for every single token across the entire internet. Any tiny bit of computation, when multiplied by a base that massive, inevitably skyrockets into astronomical numbers.
It makes me very sad to see this pseudo-intellectualism posted here and so many people replying here about consciousness and so on, not realizing what it would entail if this were true.
For LLMs to have consciousness we would approach fictional levels of how the universe works, and magical levels of how any interpretation of information as an equivalent of some qualia would magically apply. (E.G. the word hurt in output by an LLM, would be associated with pain)
You can't deduce consciousness or qualia from the output of an LLM.
Sure on a purely philosophical level, since qualia isn't measurable, you can claim that it can exist in anything, even inanimate objects, but this argument is as moot as anything that approaches the limits of philosophy.
But overall, there is no reason to believe LLMs have qualia or consciousness, it would be absolutely absurd.
This would imply that information in itself would magically entail qualia based on it's valance or something like that.
An LLM "saying" I am in pain, won't magically make the pain appear, based on what criteria? Even algorithmically there is no basis to even simulate something like this, it is impossible for it to emerge architecturally.
Humans don't feel pain because on a purely information level this is negative for the organism, obviously the nervous system does something deliberate to signal pain, and it evolved this way.
And also don't forget the dynamic aspects of the brain, and the binding problem, consciousness and qualia can't exist statically, you can't have a gpu (or piece of paper) represent a computation or w/e and qualia to exist.
The binding problem itself entails that the brain is doing something in particular to solve it, I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information.
If it were otherwise, then it would go into magical territory, it would mean the information itself would raise to qualia, and it would also entail that you wouldn't even need physical connections between neurons, just for them to behave this way and represent information. E.G. replace each neuron with a microscopic led or w/e, and each synapse with radio waves or w/e, if qualia didn't have a physical aspect, and was purely informational and computational then this would imply that you can ultimately derive it from something as abstract as numbers on a piece of paper, and when you get to that point, you not only can't solve the binding problem, and it becomes magical, but you also can't solve the valance/direction problem, it would imply that something like pain, or any negative or positive sensation arises purely from the interpretation aspect of the information, but we know this isn't the case, organism evolved to represent in particular such signals, for survival for example
Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.
Some questions:
1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?
2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?
3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?
4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?
And a further question - a dichotomy:
A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?
B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?
My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.
However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.
No mention of ‘static’ vs. ‘dynamic’ is a bit disappointing in reference to the weights. Because you could argue that every neuron in your nervous system can be modeled as a collection of weights, firing likelihoods, receptor sensitivities, current dynamic state of that neuron - but LLMs are static collections of weights at inference time, with the dynamic adjustment of weights takes place at training time. So, just a ROM construct, like something out of Neuromancer, just trained on all written knowledge, not just one person’s total lived experience.
The above take fails in the real world because neuronal cells don’t exist in a vacuum; they are products of cellular development from a zygotic union of haploid contributors of sequential genetic information optimized for survival in an oxygen-rich biosphere powered largely by our local star that supports mammalian life (and microbial, plant, avian, etc.). Real AI would thus be AL - artificial life - as much as artificial intelligence. I don’t think you can have the one without the other, which upsets the simulationists who think an agent in the Matrix would be intelligent.
What either interpretation implies is that any real ‘artificial’ intelligence would be no more artificial than you or I, but it would have to dynamically update its weights at the same speed a human nervous system could (think how quickly we learn not to poke a cactus). For it to be at all trustworthy, then like a human, it would have to undergo a socialization process, one of the results of which is the development of a sense of embarrassment when it breaks acceptable social norms.
Hmm, this reminds me of the recent statement of the Pope about AI, of which I immediately thought, “Wait a second, aren’t there a fair number of people like this? The narcissistic sociopath profile, I think it’s called, a bit unfair to assume any real AI would turn out this way, isn’t it?”
Pope: “ Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate or even simulate, but they do not understand what they produce, for they lack the affective, relational, and spiritual perspective through which human beings grow in wisdom.”
They are semiotic infrastructure frozen in a state. We shouldn't keep pretending this is cognitive and using cognitive terms to frame. It’s incredibly stupid. Sorry to inform all of us computer scientist that semiotics has your milk.