I have the technical knowledge to know how LLMs work, but I still find it pointless to not anthropomorphize, at least to an extent.
The language of "generator that stochastically produces the next word" is just not very useful when you're talking about, e.g., an LLM that is answering complex world modeling questions or generating a creative story. It's at the wrong level of abstraction, just as if you were discussing an UI events API and you were talking about zeros and ones, or voltages in transistors. Technically fine but totally useless to reach any conclusion about the high-level system.
We need a higher abstraction level to talk about higher level phenomena in LLMs as well, and the problem is that we have no idea what happens internally at those higher abstraction levels. So, considering that LLMs somehow imitate humans (at least in terms of output), anthropomorphization is the best abstraction we have, hence people naturally resort to it when discussing what LLMs can do.
On the contrary, anthropomorphism IMO is the main problem with narratives around LLMs - people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort (actively encouraged by the companies selling them) and it is completely distorting discussions on their use and perceptions of their utility.
I kinda agree with both of you. It might be a required abstraction, but it's a leaky one.
Long before LLMs, I would talk about classes / functions / modules like "it then does this, decides the epsilon is too low, chops it up and adds it to the list".
The difference I guess it was only to a technical crowd and nobody would mistake this for anything it wasn't. Everybody know that "it" didn't "decide" anything.
With AI being so mainstream and the math being much more elusive than a simple if..then I guess it's just too easy to take this simple speaking convention at face value.
Agreeing with you, this is a "can a submarine swim" problem IMO. We need a new word for what LLMs are doing. Calling it "thinking" is stretching the word to breaking point, but "selecting the next word based on a complex statistical model" doesn't begin to capture what they're capable of.
What does a submarine do? Submarine? I suppose you "drive" a submarine which is getting to the idea: submarines don't swim because ultimately they are "driven"? I guess the issue is we don't make up a new word for what submarines do, we just don't use human words.
I think the above poster gets a little distracted by suggesting the models are creative which itself is disputed. Perhaps a better term, like above, would be to just use "model". They are models after all. We don't make up a new portmanteau for submarines. They float, or drive, or submarine around.
So maybe an LLM doesn't "write" a poem, but instead "models a poem" which maybe indeed take away a little of the sketchy magic and fake humanness they tend to be imbued with.
We can argue all day what "think" means and whether a LLM thinks (probably not IMO), but at least in my head the threshold for "decide" is much lower so I can perfectly accept that a LLM (or even a class) "decides". I don't have a conflict about that. Yeah, it might not be a decision in the human sense, but it's a decision in the mathematical sense so I have always meant "decide" literally when I was talking about a piece of code.
It's much more interesting when we are talking about... say... an ant... Does it "decide"? That I have no idea as it's probably somewhere in between, neither a sentient decision, nor a mathematical one.
When I see these debates it's always the other way around - one person speaks colloquially about an LLM's behavior, and then somebody else jumps on them for supposedly believing the model is conscious, just because the speaker said "the model thinks.." or "the model knows.." or whatever.
To be honest the impression I've gotten is that some people are just very interested in talking about not anthropomorphizing AI, and less interested in talking about AI behaviors, so they see conversations about the latter as a chance to talk about the former.
Respectfully, that is a reflection of the places you hang out in (like HN) and not the reality of the population.
Outside the technical world it gets much worse. There are people who killed themselves because of LLMs, people who are in love with them, people who genuinely believe they have “awakened” their own private ChatGPT instance into AGI and are eschewing the real humans in their lives.
Anthropomorphising implicitly assumes motivation, goals and values. That's what the core of anthropomorphism is - attempting to explain behavior of a complex system in teleological terms. And prompt escapes make it clear LLMs doesn't have any teleological agency yet. Whenever their course of action is, it is to easy to steer them of. Try to do it with a sufficiently motivated human.
I think it's worth distinguishing between the use of anthropomorphism as a useful abstraction and the misuse by companies to fuel AI hype.
For example, I think "chain of thought" is a good name for what it denotes. It makes the concept easy to understand and discuss, and a non-antropomorphized name would be unnatural and unnecessarily complicate things. This doesn't mean that I support companies insisting that LLMs think just like humans or anything like that.
By the way, I would say actually anti-anthropomorphism has been a bigger problem for understanding LLMs than anthropomorphism itself. The main proponents of anti-anthropomorphism (e.g. Bender and the rest of "stochastic parrot" and related paper authors) came up with a lot of predictions about things that LLMs surely couldn't do (on account of just being predictors of the next word, etc.) which turned out to be spectacularly wrong.
I don't know about others, but I much prefer if some reductionist tries to conclude what's technically feasible and is proven wrong over time, than somebody yelling holistic analogies á la "it's sentient, it's intelligent, it thinks like us humans" for the sole dogmatic reason of being a futurist.
Tbh I also think your comparison that puts "UI events -> Bits -> Transistor Voltages" as analogy to "AI thinks -> token de-/encoding + MatMul" is certainly a stretch, as the part about "Bits -> Transistor Voltages" applies to both hierarchies as the foundational layer.
"chain of thought" could probably be called "progressive on-track-inference" and nobody would roll an eye.
I get the impression after using language models for quite a while that perhaps the one thing that is riskiest to anthropomorphise is the conversational UI that has become the default for many people.
A lot of the issues I'd have when 'pretending' to have a conversation are much less so when I either keep things to a single Q/A pairing, or at the very least heavily edit/prune the conversation history. Based on my understanding of LLM's, this seems to make sense even for the models that are trained for conversational interfaces.
so, for example, an exchange with multiple messages, where at the end I ask the LLM to double-check the conversation and correct 'hallucinations', is less optimal than something like asking for a thorough summary at the end, and then feeding that into a new prompt/conversation, as the repetition of these falsities, or 'building' on them with subsequent messages, is more likely to make them a stronger 'presence' and as a result perhaps affect the corrections.
I haven't tested any of this thoroughly, but at least with code I've definitely noticed how a wrong piece of code can 'infect' the conversation.
This is why I actually really love the description of it as a "Shoggoth" - it's more abstract, slightly floaty but it achieves the purpose of not treating and anthropomising it as a human being while not treating LLMs as a collection of predictive words.
You are conflating anthropomorphism with personification. They are not the same thing. No one believes their guitar or car or boat is alive and sentient when they give it a name or talk to or about it.
But the author used "anthropomorphism" the same way as I did. I guess we both mean "personification" then.
> we talk about "behaviors", "ethical constraints", and "harmful actions in pursuit of their goals". All of these are anthropocentric concepts that - in my mind - do not apply to functions or other mathematical objects.
One talking about a program's "behaviors", "actions" or "goals" doesn't mean they believe the program is sentient. Only "ethical constraints" is suspiciously anthropomorphizing.
I'm not convinced... we use these terms to assign roles, yes, but these roles describe a utility or assign a responsibility. That isn't anthropomorphizing anything, but it rather describes the usage of an inanimate object as tool for us humans and seems in line with history.
What's the utility or the responsibility of AI, what's its usage as tool? If you'd ask me it should be closer to serving insights than "reasoning thoughts".
The "point" of not anthropomorphizing is to refrain from judgement until a more solid abstraction appears. The problem with explaining LLMs in terms of human behaviour is that, while we don't clearly understand what the LLM is doing, we understand human cognition even less! There is literally no predictive power in the abstraction "The LLM is thinking like I am thinking". It gives you no mechanism to evaluate what tasks the LLM "should" be able to do.
Seriously, try it. Why don't LLMs get frustrated with you if you ask them the same question repeatedly? A human would. Why are LLMs so happy to give contradictory answers, as long as you are very careful not to highlight the contradictory facts? Why do earlier models behave worse on reasoning tasks than later ones? These are features nobody, anywhere understands. So why make the (imo phenomenally large) leap to "well, it's clearly just a brain"?
It is like someone inventing the aeroplane and someone looks at it and says "oh, it's flying, I guess it's a bird". It's not a bird!
> Why don't LLMs get frustrated with you if you ask them the same question repeatedly?
To be fair, I have had a strong sense of Gemini in particular becoming a lot more frustrated with me than GPT or Claude.
Yesterday I had it ensuring me that it was doing a great job, it was just me not understanding the challenge but it would break it down step by step just to make it obvious to me (only to repeat the same errors, but still)
I’ve just interpreted it as me reacting to the lower amount of sycophancy for now
The author seems to want to label any discourse as “anthropomorphizing”. The word “goal” stood out to me: the author wants us to assume that we're anthropomorphizing as soon as we even so much as use the word “goal”. A simple breadth-first search that evaluates all chess boards and legal moves, but stops when it finds a checkmate for white and outputs the full decision tree, has a “goal”. There is no anthropomorphizing here, it's just using the word “goal” as a technical term. A hypothetical AGI with a goal like paperclip maximization is just a logical extension of the breadth-first search algorithm. Imagining such an AGI and describing it as having a goal isn't anthropomorphizing.
The problem with viewing LLMs as just sequence generators, and malbehaviour as bad sequences, is that it simplifies too much. LLMs have hidden state not necessarily directly reflected in the tokens being produced and it is possible for LLMs to output tokens in opposition to this hidden state to achieve longer term outcomes (or predictions, if you prefer).
Is it too anthropomorphic to say that this is a lie? To say that the hidden state and its long term predictions amount to a kind of goal? Maybe it is. But we then need a bunch of new words which have almost 1:1 correspondence to concepts from human agency and behavior to describe the processes that LLMs simulate to minimize prediction loss.
Reasoning by analogy is always shaky. It probably wouldn't be so bad to do so. But it would also amount to impenetrable jargon. It would be an uphill struggle to promulgate.
Instead, we use the anthropomorphic terminology, and then find ways to classify LLM behavior in human concept space. They are very defective humans, so it's still a bit misleading, but at least jargon is reduced.
I think that the hidden state is really just at work improving the model's estimation of the joint probability over tokens. And the assumption here, which failed miserably in the early 20th century in the work of the logical posivitists, is that if you can so expertly estimate that joint probability of language, then you will be able to understand "knowledge." But there's no well grounded reason to believe that and plenty of the reasons (see: the downfall of logical posivitism) to think that language is an imperfect representation of knowledge. In other words, what humans do when we think is more complicated than just learning semiotic patterns and regurgitating them. Philosophical skeptics like Hume thought so, but most epistemology writing after that had better answers for how we know things.
IMHO, anthrophormization of LLMs is happening because it's perceived as good marketing by big corporate vendors.
People are excited about the technology and it's easy to use the terminology the vendor is using. At that point I think it gets kind of self fulfilling. Kind of like the meme about how to pronounce GIF.
IMHO it happens for the same reason we see shapes in clouds. The human mind through millions of years has evolved to equate and conflate the ability to generate cogent verbal or written output with intelligence. It's an instinct to equate the two. It's an extraordinarily difficult instinct to break. LLMs are optimised for the one job that will make us confuse them for being intelligent
I think anthropomorphizing LLMs is useful, not just a marketing tactic. A lot of intuitions about how humans think map pretty well to LLMs, and it is much easier to build intuitions about how LLMs work by building upon our intuitions about how humans think than by trying to build your intuitions from scratch.
Would this question be clear for a human? If so, it is probably clear for an LLM. Did I provide enough context for a human to diagnose the problem? Then an LLM will probably have a better chance of diagnosing the problem. Would a human find the structure of this document confusing? An LLM would likely perform poorly when reading it as well.
Re-applying human intuitions to LLMs is a good starting point to gaining intuition about how to work with LLMs. Conversely, understanding sequences of tokens and probability spaces doesn't give you much intuition about how you should phrase questions to get good responses from LLMs. The technical reality doesn't explain the emergent behaviour very well.
I don't think this is mutually exclusive with what the author is talking about either. There are some ways that people think about LLMs where I think the anthropomorphization really breaks down. I think the author says it nicely:
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost.
We are making user interfaces. Good user interfaces are intuitive and purport to be things that users are familiar with, such as people. Any alternative explanation of such a versatile interface will be met with blank stares. Users with no technical expertise would come to their own conclusions, helped in no way by telling the user not to treat the chat bot as a chat bot.
Nobody cares about what’s perceived as good marketing. People care about what resonates with the target market.
But yes, anthropomorphising LLMs is inevitable because they feel like an entity. People treat stuffed animals like creatures with feelings and personality; LLMs are far closer than that.
the chat interface was a choice, though a natural one. before they'd RLHFed it into chatting and it was just GPT 3 offering completions 1) not very many people used it and 2) it was harder to anthropomorphize
> People treat stuffed animals like creatures with feelings and personality; LLMs are far closer than that.
Children do, some times, but it's a huge sign of immaturity when adults, let alone tech workers, do it.
I had a professor at University that would yell at us if/when we personified/anthropomorphized the tech, and I have that same urge when people ask me "What does <insert LLM name here> think?".
Do they ?
LLM embedd the token sequence N^{L} to R^{LxD}, we have some attention and the output is also R^{LxD}, then we apply a projection to the vocabulary and we get R^{LxV} we get therefore for each token a likelihood over the voc.
In the attention, you can have Multi Head attention (or whatever version is fancy: GQA,MLA) and therefore multiple representation, but it is always tied to a token. I would argue that there is no hidden state independant of a token.
Whereas LSTM, or structured state space for example have a state that is updated and not tied to a specific item in the sequence.
I would argue that his text is easily understandable except for the notation of the function, explaining that you can compute a probability based on previous words is understandable by everyone without having to resort to anthropomorphic terminology
There is hidden state as plain as day merely in the fact that logits for token prediction exist. The selected token doesn't give you information about how probable other tokens were. That information, that state which is recalculated in autoregression, is hidden. It's not exposed. You can't see it in the text produced by the model.
There is plenty of state not visible when an LLM starts a sentence that only becomes somewhat visible when it completes the sentence. The LLM has a plan, if you will, for how the sentence might end, and you don't get to see an instance of that plan unless you run autoregression far enough to get those tokens.
Similarly, it has a plan for paragraphs, for whole responses, for interactive dialogues, plans that include likely responses by the user.
Arguably there's reason to believe it comes up with a plan when it is computing token propabilities, but it does not store it between tokens. I.e. it doesn't possess or "have" it. It simply comes up with a plan, emits a token, and entirely throws all its intermediate thoughts (including any plan) to start again from scratch on the next token.
It's true that the last layer's output for a given input token only affects the corresponding output token and is discarded afterwards. But the penultimate layer's output affects the computation of the last layer for all future tokens, so it is not discarded, but stored (in the KV cache). Similarly for the antepenultimate layer affecting the penultimate layer and so on.
So there's plenty of space in intermediate layers to store a plan between tokens without starting from scratch every time.
I believe saying the LLM has a plan is a useful anthropomorphism for the fact that it does have hidden state that predicts future tokens, and this state conditions the tokens it produces earlier in the stream.
I don't think that the comment above you made any suggestion that the plan is persisted between token generations. I'm pretty sure you described exactly what they intended.
- the sufficient amount of information to do evolution of the system. The state of a pendulum is it's position and velocity (or momentum). If you take a single picture of a pendulum, you do not have a representation that lets you make predictions.
- information that is persisted through time. A stateful protocol is one where you need to know the history of the messages to understand what will happen next. (Or, analytically, it's enough to keep track of the sufficient state.) A procedure with some hidden state isn't a pure function. You can make it a pure function by making the state explicit.
this sounds like a fun research area. do LLMs have plans about future tokens?
how do we get 100 tokens of completion, and not just one output layer at a time?
are there papers youve read that you can share that support the hypothesis? vs that the LLM doesnt have ideas about the future tokens when its predicting the next one?
I'm not sure what you mean by "hidden state". If you set aside chain of thought, memories, system prompts, etc. and the interfaces that don't show them, there is no hidden state.
These LLMs are almost always, to my knowledge, autoregressive models, not recurrent models (Mamba is a notable exception).
Hidden state in the form of the activation heads, intermediate activations and so on. Logically, in autoregression these are recalculated every time you run the sequence to predict the next token. The point is, the entire NN state isn't output for each token. There is lots of hidden state that goes into selecting that token and the token isn't a full representation of that information.
Hidden layer is a term of art in machine learning / neural network research. See https://en.wikipedia.org/wiki/Hidden_layer . Somehow this term mutated into "hidden state", which in informal contexts does seem to be used quite often the way the grandparent comment used it.
That's not what "state" means, typically. The "state of mind" you're in affects the words you say in response to something.
Intermediate activations isn't "state". The tokens that have already been generated, along with the fixed weights, is the only data that affects the next tokens.
Sure it's state. It logically evolves stepwise per token generation. It encapsulates the LLM's understanding of the text so far so it can predict the next token. That it is merely a fixed function of other data isn't interesting or useful to say.
All deterministic programs are fixed functions of program code, inputs and computation steps, but we don't say that they don't have state. It's not a useful distinction for communicating among humans.
I'll say it once more: I think it is useful to distinguish between autoregressive and recurrent architectures. A clear way to make that distinction is to agree that the recurrent architecture has hidden state, while the autoregressive one does not. A recurrent model has some point in a space that "encapsulates its understanding". This space is "hidden" in the sense that it doesn't correspond to text tokens or any other output. This space is "state" in the sense that it is sufficient to summarize the history of the inputs for the sake of predicting the next output.
When you use "hidden state" the way you are using it, I am left wondering how you make a distinction between autoregressive and recurrent architectures.
No, that's not quite what I mean. I used the logits in another reply to point out that there is data specific to the generation process that is not available from the tokens, but there's also the network activations adding up to that state.
Processing tokens is a bit like ticks in a CPU, where the model weights are the program code, and tokens are both input and output. The computation that occurs logically retains concepts and plans over multiple token generation steps.
That it is fully deterministic is no more interesting than saying a variable in a single threaded program is not state because you can recompute its value by replaying the program with the same inputs. It seems to me that this uninteresting distinction is the GP's issue.
If you dont know, that's not necessarily anyone's fault, but why are you dunking into the conversation? The hidden state is a foundational part of a transformers implementation. And because we're not allowed to use metaphors because that is too anthropomorphic, then youre just going to have to go learn the math.
The comment you are replying to is not claiming ignorance of how models work. It is saying that the author does know how they work, and they do not contain anything that can properly be described as "hidden state". The claimed confusion is over how the term "hidden state" is being used, on the basis that it is not being used correctly.
I don't think your response is very productive, and I find that my understanding of LLMs aligns with the person you're calling out. We could both be wrong, but I'm grateful that someone else spoke saying that it doesn't seem to match their mental model and we would all love to learn a more correct way of thinking about LLMs.
Telling us to just go and learn the math is a little hurtful and doesn't really get me any closer to learning the math. It gives gatekeeping.
do LLM models consider future tokens when making next token predictions?
eg. pick 'the' as the next token because there's a strong probability of 'planet' as the token after?
is it only past state that influences the choice of 'the'? or that the model is predicting many tokens in advance and only returning the one in the output?
if it does predict many, id consider that state hidden in the model weights.
The most obvious case of this is in terms of `an apple` vs `a pear`. LLMs never get the a-an distinction wrong, because their internal state 'knows' the word that'll come next.
If I give an LLM a fragment of text that starts with, "The fruit they ate was an <TOKEN>", regardless of any plan, the grammatically correct answer is going to force a noun starting with a vowel. How do you disentangle the grammar from planning?
Going to be a lot more "an apple" in the corpus than "an pear"
> Is it too anthropomorphic to say that this is a lie?
Yes. Current LLMs can only introspect from output tokens. You need hidden reasoning that is within the black box, self-knowing, intent, and motive to lie.
I rather think accusing an LLM of lying is like accusing a mousetrap of being a murderer.
When models have online learning, complex internal states, and reflection, I might consider one to have consciousness and to be capable of lying. It will need to manifest behaviors that can only emerge from the properties I listed.
I've seen similar arguments where people assert that LLMs cannot "grasp" what they are talking about. I strongly suspect a high degree of overlap between those willing to anthropomorphize error bars as lies while declining to award LLMs "grasping". Which is it? It can think or it cannot? (objectively, SoTA models today cannot yet.) The willingness to waffle and pivot around whichever perspective damns the machine completely belies the lack of honesty in such conversations.
> Current LLMs can only introspect from output tokens
The only interpretation of this statement I can come up with is plain wrong. There's no reason LLM shouldn't be able to introspect without any output tokens. As the GP correctly says, most of the processing in LLMs happens over hidden states. Output tokens are just an artefact for our convenience, which also happens to be the way the hidden state processing is trained.
The recurrence comes from replaying tokens during autoregression.
It's as if you have a variable in a deterministic programming language, only you have to replay the entire history of the program's computation and input to get the next state of the machine (program counter + memory + registers).
Producing a token for an LLM is analogous to a tick of the clock for a CPU. It's the crank handle that drives the process.
But the function of an unrolled recursion is the same as a recursive function with bounded depth as long as the number of unrolled steps match. The point is whatever function recursion is supposed to provide can plausibly be present in LLMs.
> Output tokens are just an artefact for our convenience
That's nonsense. The hidden layers are specifically constructed to increase the probability that the model picks the right next word. Without the output/token generation stage the hidden layers are meaningless. Just empty noise.
It is fundamentally an algorithm for generating text. If you take the text away it's just a bunch of fmadds. A mute person can still think, an LLM without output tokens can do nothing.
So the author’s core view is ultimately a Searle-like view: a computational, functional, syntactic rules based system cannot reproduce a mind. Plenty of people will agree, plenty of people will disagree, and the answer is probably unknowable and just comes down to whatever axioms you subscribe to in re: consciousness.
The author largely takes the view that it is more productive for us to ignore any anthropomorphic representations and focus on the more concrete, material, technical systems - I’m with them there… but only to a point. The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like. So even if it is a stochastic system following rules, clearly the rules are complex enough (to the tune of billions of operations, with signals propagating through some sort of resonant structure, if you take a more filter impulse response like view of a sequential matmuls) to result in emergent properties. Even if we (people interested in LLMs with at least some level of knowledge of ML mathematics and systems) “know better” than to believe these systems to possess morals, ethics, feelings, personalities, etc, the vast majority of people do not have any access to meaningful understanding of the mathematical, functional representation of an LLM and will not take that view, and for all intents and purposes the systems will at least seem to have those anthropomorphic properties, and so it seems like it is in fact useful to ask questions from that lens as well.
In other words, just as it’s useful to analyze and study these things as the purely technical systems they ultimately are, it is also, probably, useful to analyze them from the qualitative, ephemeral, experiential perspective that most people engage with them from, no?
> The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like.
What you identify as emergent and mind-like is a direct result of these tools being able to mimic human communication patterns unlike anything we've ever seen before. This capability is very impressive and has a wide range of practical applications that can improve our lives, and also cause great harm if we're not careful, but any resemblance to intelligence is an illusion. An illusion that many people in this industry obsessively wish to propagate, because thar be gold in them hills.
> The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like.
For people who have only a surface-level understanding of how they work, yes. A nuance of Clarke's law that "any sufficiently advanced technology is indistinguishable from magic" is that the bar is different for everybody and the depth of their understanding of the technology in question. That bar is so low for our largely technologically-illiterate public that a bothersome percentage of us have started to augment and even replace religious/mystical systems with AI powered godbots (LLMs fed "God Mode"/divination/manifestation prompts).
Nah, as a person that knows in detail how LLMs work with probably unique alternative perspective in addition to the commonplace one, I found any claims of them not having emergent behaviors to be of the same fallacy as claiming that crows can't be black because they have DNA of a bird.
Thank you for a well thought out and nuanced view in a discussion where so many are clearly fitting arguments to foregone, largely absolutist, conclusions.
It’s astounding to me that so much of HN reacts so emotionally to LLMs, to the point of denying there is anything at all interesting or useful about them. And don’t get me started on the “I am choosing to believe falsehoods as a way to spite overzealous marketing” crowd.
Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics?
LLMs reflect (and badly I may add) aspects of the human thought process. If you take a leap and say they are anything more than that, you might as well start considering the person appearing in your mirror as a living being.
Literally (and I literally mean it) there is no difference. The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it. Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.
> Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics?
We know that Newton's laws are wrong, and that you have to take special and general relativity into account. Why would we ever teach anyone Newton's laws any more?
I don’t mean to amplify a false understanding at all. I probably did not articulate myself well enough, so I’ll try again.
I think it is inevitable that some - many - people will come to the conclusion that these systems have “ethics”, “morals,” etc, even if I or you personally do not think they do. Given that many people may come to that conclusion though, regardless of if the systems do or do not “actually” have such properties, I think it is useful and even necessary to ask questions like the following: “if someone engages with this system, and comes to the conclusion that it has ethics, what sort of ethics will they be likely to believe the system has? If they come to the conclusion that it has ‘world views,’ what ‘world views’ are they likely to conclude the system has, even if other people think it’s nonsensical to say it has world views?”
> The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it.
Surely this is not quite accurate - the material properties - surface roughness, reflectivity, geometry, etc - all influence the appearance of a perceptible image of a person. Look at yourself in a dirty mirror, a new mirror, a shattered mirror, a funhouse distortion mirror, a puddle of water, a window… all of these produce different images of a person with different attendant phenomenological experiences of the person seeing their reflection. To take that a step further - the entire practice of portrait photography is predicated on the idea that the collision of different technical systems with the real world can produce different semantic experiences, and it’s the photographer’s role to tune and guide the system to produce some sort of contingent affect on the person viewing the photograph at some point in the future. No, there is no “real” person in the photograph, and yet, that photograph can still convey something of person-ness, emotion, memory, etc etc. This contingent intersection of optics, chemical reactions, lighting, posture, etc all have the capacity to transmit something through time and space to another person. It’s not just a meaningless arrangement of chemical structures on paper.
> Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.
But, we are feeding it with such data artifacts and will likely continue to do so for a while, and so it seems reasonable to ask what it is “reflecting” back…
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
This is such a bizarre take.
The relation associating each human to the list of all words they will ever say is obviously a function.
> almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities.
There's a rich family of universal approximation theorems [0]. Combining layers of linear maps with nonlinear cutoffs can intuitively approximate any nonlinear function in ways that can be made rigorous.
The reason LLMs are big now is that transformers and large amounts of data made it economical to compute a family of reasonably good approximations.
> The following is uncomfortably philosophical, but: In my worldview, humans are dramatically different things than a function . For hundreds of millions of years, nature generated new versions, and only a small number of these versions survived.
This is just a way of generating certain kinds of functions.
Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to.
Not that this is your main point, but I find this take representative, “do you believe there's anything about humans that exists outside the mathematical laws of physics?”There are things “about humans”, or at least things that our words denote, that are outside physic’s explanatory scope. For example, the experience of the colour red cannot be known, as an experience, by a person who only sees black and white. This is the case no matter what empirical propositions, or explanatory system, they understand.
Perhaps. But I can't see a reason why they couldn't still write endless—and theoretically valuable—poems, dissertations, or blog posts, about all things red and the nature of redness itself. I imagine it would certainly take some studying for them, likely interviewing red-seers, or reading books about all things red. But I'm sure they could contribute to the larger red discourse eventually, their unique perspective might even help them draw conclusions the rest of us are blind to.
So perhaps the fact that they "cannot know red" is ultimately irrelevant for an LLM too?
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside, whether due to religion or philosophy or whatever, and suggesting that they just not do that.
In my experience, that's not a particularly effective tactic.
Rather, we can make progress by assuming their predicate: Sure, it's a room that translates Chinese into English without understanding, yes, it's a function that generates sequences of words that's not a human... but you and I are not "it" and it behaves rather an awful lot like a thing that understands Chinese or like a human using words. If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
Conversely, when speaking with such a person about the nature of humans, we'll have to agree to dismiss the elements that are different from a function. The author says:
> In my worldview, humans are dramatically different things than a function... In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not". Maybe that human has a unique understanding of the nature of that particular piece of pop culture artwork, maybe it makes them feel things that an LLM cannot feel in a part of their consciousness that an LLM does not possess. But for the purposes of the question, we're merely concerned with whether a human or LLM will generate a particular sequence of words.
>> given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
> Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not".
I think you may have this flipped compared to what the author intended. I believe the author is not talking about the probability of an output given an input, but the probability of a given output across all inputs.
Note that the paragraph starts with "In my worldview, humans are dramatically different things than a function, (R^n)^c -> (R^n)^c". To compute a probability of a given output, (which is a any given element in "(R^n)^n"), we can count how many mappings there are total and then how many of those mappings yield the given element.
The point I believe is to illustrate the complexity of inputs for humans. Namely for humans the input space is even more complex than "(R^n)^c".
In your example, we can compute how many input phrases into a LLM would produce the output "make it or not". We can than compute that ratio to all possible input phrases. Because "(R^n)^c)" is finite and countable, we can compute this probability.
For a human, how do you even start to assess the probability that a human would ever say "make it or not?" How do you even begin to define the inputs that a human uses, let alone enumerate them? Per the author, "We understand essentially nothing about it." In other words, the way humans create their outputs is (currently) incomparably complex compared to a LLM, hence the critique of the anthropomorphization.
I see your point, and I like that you're thinking about this from the perspective of how to win hearts and minds.
I agree my approach is unlikely to win over the author or other skeptics. But after years of seeing scientists waste time trying to debate creationists and climate deniers I've kind of given up on trying to convince the skeptics. So I was talking more to HN in general.
> You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside
I'm not sure what it means to be observable or not from the outside. I think this is at least partially because I don't know what it means to be inside either. My point was just that whatever consciousness is, it takes place in the physical world and the laws of physics apply to it. I mean that to be as weak a claim as possible: I'm not taking any position on what consciousness is or how it works etc.
Searle's Chinese room argument attacks attacks a particular theory about the mind based essentially turing machines or digital computers. This theory was popular when I was in grad school for psychology. Among other things, people holding the view that Searle was attacking didn't believe that non-symbolic computers like neural networks could be intelligent or even learn language. I thought this was total nonsense, so I side with Searle in my opposition to it. I'm not sure how I feel about the Chinese room argument in particular, though. For one thing it entirely depends on what it means to "understand" something, and I'm skeptical that humans ever "understand" anything.
> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
I see what you're saying: that a technically incorrect assumption can bring to bear tools that improve our analysis. My nitpick here is I agree with OP that we shouldn't anthropomorphize LLMs, any more than we should anthropomorphize dogs or cats. But OP's arguments weren't actually about anthropomorphizing IMO, they were about things like functions that are more fundamental than humans. I think artificial intelligence will be non-human intelligence just like we have many examples of non-human intelligence in animals. No attribution of human characteristics needed.
> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
Yes I agree with you about your lyrics example. But again here I think OP is incorrect to focus on the token generation argument. We all agree human speech generates tokens. Hopefully we all agree that token generation is not completely predictable. Therefore it's by definition a randomized algorithm and it needs to take an RNG. So pointing out that it takes an RNG is not a valid criticism of LLMs.
Unless one is a super-determinist then there's randomness at the most basic level of physics. And you should expect that any physical process we don't understand well yet (like consciousness or speech) likely involves randomness. If one *is* a super-determinist then there is no randomness, even in LLMs and so the whole point is moot.
>Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to.
It seems like, we can at best, claim that we have modeled the human thought process for reasoning/analytic/quantitative through Linear Algebra, as the best case. Why should we expect the model to be anything more than a model ?
I understand that there is tons of vested interest, many industries, careers and lives literally on the line causing heavy bias to get to AGI. But what I don't understand is what about linear algebra that makes it so special that it creates a fully functioning life or aspects of a life?
Should we make an argument saying that Schroedinger's cat experiment can potentially create zombies then the underlying Applied probabilistic solutions should be treated as super-human and build guardrails against it building zombie cats?
>Why should we expect the model to be anything more than a model ?
To model a process with perfect accuracy requires recovering the dynamics of that process. The question we must ask is what happens in the space between bad statistical model and perfect accuracy? What happens when the model begins to converge towards accurate reproduction. How far does generalization in the model take us towards capturing the dynamics involved in thought?
> It seems like, we can at best, claim that we have modeled the human thought process for reasoning/analytic/quantitative through Linear Algebra....I don't understand is what about linear algebra that makes it so special that it creates a fully functioning life or aspects of a life?
Not linear algebra. Artificial neural networks create arbitrarily non-linear functions. That's the point of non-linear activation functions and it's the subject of the universal approximation theorems I mentioned above.
ANNs are just mathematical transformations, powered by linear algebra + non-linear functions.
They simulate certain cognitive processes — but they are fundamentally math, not magic.
People anthropomorphize just about anything around them. People talk about inanimate objects like they are persons. Ships, cars, etc. And of course animals are well in scope for this as well, even the ones that show little to no signs of being able to reciprocate the relationship (e.g. an ant). People talk to their plants even.
It's what we do. We can't help ourselves. There's nothing crazy about it and most people are perfectly well aware that their car doesn't love them back.
LLMs are not conscious because unlike human brains they don't learn or adapt (yet). They basically get trained and then they become read only entities. So, they don't really adapt to you over time. Even so, LLMs are pretty good and can fake a personality pretty well. And with some clever context engineering and alignment, they've pretty much made the Turing test irrelevant; at least over the course of a short conversation. And they can answer just about any question in a way that is eerily plausible from memory, and with the help of some tools actually pretty damn good for some of the reasoning models.
Anthropomorphism was kind of a foregone conclusion the moment we created computers; or started thinking about creating one. With LLMs it's pretty much impossible not to anthropomorphize. Because they've actually been intentionally imitate human communication. That doesn't mean that we've created AGIs yet. For that we need some more capability. But at the same time, the learning processes that we use to create LLMs are clearly inspired by how we learn ourselves. Our understanding of how that works is far from perfect but it's yielding results. From here to some intelligent thing that is able to adapt and learn transferable skills is no longer unimaginable.
The short term impact is that LLMs are highly useful tools that have an interface that is intentionally similar to how we'd engage with others. So we can talk and it listens. Or write and it understands. And then it synthesizes some kind of response or starts asking questions and using tools. The end result is quite a bit beyond what we used to be able to expect from computers. And it does not require a lot of training of people to be able to use them.
> LLMs are not conscious because unlike human brains they don't learn or adapt (yet).
That's neither a necessary nor sufficient condition.
In order to be conscious, learning may not be needed, but a perception of the passing of time may be needed which may require some short-term memory. People with severe dementia often can't even remember the start of a sentence they are reading, they can't learn, but they are certainly conscious because they have just enough short-term memory.
And learning is not sufficient either. Consciousness is about being a subject, about having a subjective experience of "being there" and just learning by itself does not create this experience. There is plenty of software that can do some form of real-time learning but it doesn't have a subjective experience.
I highly recommend playing with embeddings in order to get a stronger intuitive sense of this. It really starts to click that it's a representation of high dimensional space when you can actually see their positions within that space.
Not making a qualitative assessment of any of it. Just pointing out that there are ways to build separate sets of intuition outside of using the "usual" presentation layer. It's very possible to take a red-team approach to these systems, friend.
My question: how do we know that this is not similar to how human brains work. What seems intuitively logical to me is that we have brains evolved through evolutionary process via random mutations yielding in a structure that has its own evolutionary reward based algorithms designing it yielding a structure that at any point is trying to predict next actions to maximise survival/procreation, of course with a lot of sub goals in between, ultimately becoming this very complex machinery, but yet should be easily simulated if there was enough compute in theory and physical constraints would allow for it.
Because, morals, values, consciousness etc could just be subgoals that arised through evolution because they support the main goals of survival and procreation.
And if it is baffling to think that a system could rise up, how do you think it is possible life and humans came to existence in the first place? How could it be possible? It is already happened from a far unlikelier and strange place. And wouldn't you think the whole World and the timeline in theory couldn't be represented as a deterministic function. And if not then why should "randomness" or anything else bring life to existence.
> how do we know that this is not similar to how human brains work.
Do you forget every conversation as soon as you have them? When speaking to another person, do they need to repeat literally everything they said and that you said, in order, for you to retain context?
If not, your brain does not work like an LLM. If yes, please stop what you’re doing right now and call a doctor with this knowledge. I hope Memento (2000) was part of your training data, you’re going to need it.
If we developed feelings, morals and motivation due to them being good subgoals for primary goals, survival and procreation why couldn't other systems do that. You don't have to call them the same word or the same thing, but feeling is a signal that motivates a behaviour in us, that in part has developed from generational evolution and in other part by experiences in life. There was a random mutation that made someone develop a fear signal on seeing a predator and increased the survival chances, then due to that the mutation became widespread. Similarly a feeling in a machine could be a signal it developed that goes through a certain pathway to yield in a certain outcome.
The real challenge is not to see it as a binary (the machine either has feelings or it has none). It's possible for the machine to have emergent processes or properties that resemble human feelings in their function and their complexity, but are otherwise nothing like them (structured very differently and work on completely different principles). It's possible to have a machine or algorithm so complex that the question of whether it has feelings is just a semantic debate on what you mean by “feelings” and where you draw the line.
A lot of the people who say “machines will never have feelings” are confident in that statement because they draw the line incredibly narrowly: if it ain't human, it ain't feeling. This seems to me putting the cart before the horse. It ain't feeling because you defined it so.
> My question: how do we know that this is not similar to how human brains work.
It is similar to how human brains operate. LLMs are the (current) culmination of at least 80 years of research on building computational models of the human brain.
Is it? Do we know how human brains operate? We know the basic architecture of them, so we have a map, but we don't know the details.
"The cellular biology of brains is relatively well-understood, but neuroscientists have not yet generated a theory explaining how brains work. Explanations of how neurons collectively operate to produce what brains can do are tentative and incomplete." [1]
"Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work." [1]
This is just a semantic debate on what counts as “similar”. It's possible to disagree on this point despite agreeing on everything relating to how LLMs and human brains work.
I think it's just an unfair comparison in general. The power of the LLM is the zero risk to failure, and lack of consequence when it does. Just try again, using a different prompt, retrain maybe, etc.
Humans make a bad choice, it can end said human's life. The worst choice a LLM makes just gets told "no, do it again, let me make it easier"
But an LLM model could perform poorly in tests that it is not considered and essentially means "death" for it. But begs the question at which scope should we consider an LLM to be similar to identity of a single human. Are you the same you as you were few minutes back or 10 years back? Is LLM the same LLM it is after it has been trained for further 10 hours, what if the weights are copy pasted endlessly, what if we as humans were to be cloned instantly? What if you were teleported from location A to B instantly, being put together from other atoms from elsewhere?
Ultimately this matters from evolutionary evolvement and survival of the fittest idea, but it makes the question of "identity" very complex. But death will matter because this signals what traits are more likely to keep going into new generations, for both humans and LLMs.
Death, essentially for an LLM would be when people stop using it in favour of some other LLM performing better.
The people in this thread incredulous at the assertion that they are not God and haven't invented machine life are exasperating. At this point I am convinced they, more often than not, financially benefit from their near religious position in marketing AI as akin to human intelligence.
Are we looking at the same thread? I see nobody claiming this. Anthropic does sometimes, their position is clearly wishful thinking, and it's not represented ITT.
Try looking at this from another perspective - many people simply do not see human intelligence (or life, for that matter) as magic. I see nothing religious about that, rather the opposite.
I agree with you @orbital-decay that I also do not get the same vibe reading this thread.
Though, while human intelligence is (seemingly) not magic, it is very far from being understood. The idea that a LLM is comparable to human intelligence implies that we even understand human intelligence well enough to say that.
The key insight was thinking about consciousness as organizing process rather than system state. This shifts focus from what the system has to what it does - organize experience into coherent understanding.
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost.
TFA really ought to have linked to some concrete examples of what it's disagreeing with - when I see arguments about this in practice, it's usually just people talking past each other.
Like, person A says "the model wants to X, but it knows Y is wrong, so it prefers Z", or such. And person B interprets that as ascribing consciousness or values to the model, when the speaker meant it no differently from saying "water wants to go downhill" - i.e. a way of describing externally visible behaviors, but without saying "behaves as if.." over and over.
And then in practice, an unproductive argument usually follows - where B is thinking "I am going to Educate this poor fool about the Theory of Mind", and A is thinking "I'm trying to talk about submarines; why is this guy trying to get me to argue about whether they swim?"
The author's critique of naive anthropomorphism is salient. However, the reduction to "just MatMul" falls into the same trap it seeks to avoid: it mistakes the implementation for the function. A brain is also "just proteins and currents," but this description offers no explanatory power.
The correct level of analysis is not the substrate (silicon vs. wetware) but the computational principles being executed. A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE).
To dismiss these systems as incomparable to human cognition because their form is different is to miss the point. We should not be comparing a function to a soul, but comparing the functional architectures of two different information processing systems. The debate should move beyond the sterile dichotomy of "human vs. machine" to a more productive discussion of "function over form."
> A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE).
Could you suggest some literature supporting this claim? Went through your blog post but couldn't find any.
We have no agreed-upon definition of "consciousness", no accepted understanding of what gives rise to "consciousness", no way to measure or compare "consciousness", and no test we could administer to either confirm presence of "consciousness" in something or rule it out.
The only answer to "are LLMs conscious?" is "we don't know".
It helps that the whole question is rather meaningless to practical AI development, which is far more concerned with (measurable and comparable) system performance.
This is actually not comparable, because the brain has a much more complex structure that is _not_ learned, even at that level. The proteins and their structure are not a result of training. The fixed part for LMMs is rather trivial and is, in fact, not much for than MatMul which is very easy to understand - and we do. The fixed part of the brain, including the structure of all the proteins is enormously complex which is very difficult to understand - and we don't.
It's possible to construct a similar description of whatever it is that human brain is doing that clearly fails to capture the fact that we're conscious. If you take a cross section of every nerve feeding into the human brain at a given time T, the action potentials across those cross sections can be embedded in R^n. If you take the history of those action potentials across the lifetime of the brain, you get a path through R^n that is continuous, and maps roughly onto your subjectively experienced personal history, since your brain neccesarily builds your experienced reality from this signal data moment to moment. If you then take the cross sections of every nerve feeding OUT of your brain at time T, you have another set of action potentials that can be embedded in R^m which partially determines the state of the R^n embedding at time T + delta. This is not meaningfully different from the higher dimensional game of snake described in the article, more or less reducing the experience of being a human to 'next nerve impulse prediction', but it obviously fails to capture the significance of the computation which determines what that next output should be.
I don’t see how your description “clearly fails to capture the fact that we're conscious” though.
There are many example in nature of emergent phenomena that would be very hard to predict just by looking at its components.
This is the crux of the disagreement between those that believe AGI is possible and those that don’t. Some are convinced that we “obviously” more than the sum of our parts, and thus an LLM can’t achieve consciousness because it’s missing this magic ingredient, and those that believe consciousness is just an emergent behaviour from a complex device (the brain). And thus we might be able to recreate it simply by scaling the complexity of another system.
Where exactly in my description do I invoke consciousness?
Where does the description given imply that consciousness is required in any way?
The fact that there's a non-obvious emergent phenomena which is apparently responsible for your subjective experience, and that it's possible to provide a superficially accurate description of you as a system without referencing that phenomena in any way, is my entire point. The fact that we can provide such a reductive description of LLMs without referencing consciousness has literally no bearing on whether or not they're conscious.
To be clear, I'm not making a claim as to whether they are or aren't, I'm simply pointing out that the argument in the article is fallacious.
We have a hard enough time anthropomorphizing humans! When we say he was nasty... do we know what we mean by that. Often it is "I disagree with his behaviour because..."
> A fair number of current AI luminaries have self-selected by their belief that they might be the ones getting to AGI
People in the industry, especially higher up, are making absolute bank, and it's their job to say that they're "a few years away" from AGI, regardless of if they actually believe it or not. If everyone was like "yep, we're gonna squeeze maybe 10-15% more benchie juice out of this good ole transformer thingy and then we'll have to come up with something else", I don't think that would go very well with investors/shareholders...
>I am baffled by seriously intelligent people imbuing almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities.
I am baffled by seriously intelligent people imbuing almost magical powers that can never be replicated to to something that - in my mind - is just a biological robot driven by a SNN with a bunch of hardwired stuff. Let alone attributing "human intelligence" to a single individual, when it's clearly distributed between biological evolution, social processes, and individuals.
>something that - in my mind - is just MatMul with interspersed nonlinearities
Processes in all huge models (not necessarily LLMs) can be described using very different formalisms, just like Newtonian and Lagrangian mechanics describe the same stuff in physics. You can say that an autoregressive model is a stochastic parrot that learned the input distribution, next token predictor, or that it does progressive pathfinding in a hugely multidimensional space, or pattern matching, or implicit planning, or, or, or... All of these definitions are true, but only some are useful to predict their behavior.
Given all that, I see absolutely no problem with anthropomorphizing an LLM to a certain degree, if it makes it easier to convey the meaning, and do not understand the nitpicking. Yeah, it's not an exact copy of a single Homo Sapiens specimen. Who cares.
> In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
I think that's a bit pessimistic. I think we can say for instance that the probability that a person will say "the the the of of of arpeggio halcyon" is tiny compared to the probability that they will say "I haven't been getting that much sleep lately". And we can similarly see that lots of other sequences are going to have infinitesimally low probability. Now, yeah, we can't say exactly what probability that is, but even just using a fairly sizable corpus as a baseline you could probably get a surprisingly decent estimate, given how much of what people say is formulaic.
The real difference seems to be that the manner in which humans generate sequences is more intertwined with other aspects of reality. For instance, the probability of a certain human saying "I haven't been getting that much sleep lately" is connected to how much sleep they have been getting lately. For an LLM it really isn't connected to anything except word sequences in its input.
I think this is consistent with the author's point that we shouldn't apply concepts like ethics or emotions to LLMs. But it's not because we don't know how to predict what sequences of words humans will use; it's rather because we do know a little about how to do that, and part of what we know is that it is connected with other dimensions of physical reality, "human nature", etc.
This is one reason I think people underestimate the risks of AI: the performance of LLMs lulls us into a sense that they "respond like humans", but in fact the Venn diagram of human and LLM behavior only intersects in a relatively small area, and in particular they have very different failure modes.
The author plot the input/output on a graph, intuited (largely incorrectly, because that's not how sufficiently large state spaces look) that the output was vaguely pretty, and then... I mean that's it, they just said they have a plot of the space it operates on therefore it's silly to ascribe interesting features to the way it works.
And look, it's fine, they prefer words of a certain valence, particularly ones with the right negative connotations, I prefer other words with other valences. None of this means the concerns don't matter. Natural selection on human pathogens isn't anything particularly like human intelligence and it's still very effective at selecting outcomes that we don't want against our attempts to change that, as an incidental outcome of its optimization pressures. I think it's very important we don't build highly capable systems that select for outcomes we don't want and will do so against our attempts to change it.
The anthropomorphic view of LLM is a much better representation and compression for most types of discussions and communication. A purely mathematical view is accurate but it isn’t productive for the purpose of the general public’s discourse.
I’m thinking a legal systems analogy, at the risk of a lossy domain transfer: the laws are not written as lambda calculus. Why?
And generalizing to social science and humanities, the goal shouldn’t be finding the quantitative truth, but instead understand the social phenomenon using a consensual “language” as determined by the society. And in that case, the anthropomorphic description of the LLM may gain validity and effectiveness as the adoption grows over time.
I've personally described the "stochastic parrot" model to laypeople who were worried about AI and they came away much more relaxed about it doing something "malicious". They seemed to understand the difference between "trained at roleplay" and "consciousness".
I don't think we need to simplify it to the point of considering it sentient to get the public to interact with it successfully. It causes way more problems than it solves.
Am I misunderstanding what you mean by "malicious"? It sounds like the stochastic parrot model wrongly convinced these laypeople you were talking to that they don't need to worry about LLMs doing bad things. That's definitely been my experience - the people who tell me the most about stochastic parrots are the same ones who tell me that it's absurd to worry about AI-powered disinformation or AI-powered scams.
Let's skip to the punchline. Using TFA's analogy: essentially folks are saying not that this is a set of dice rolling around making words. It's a set of dice rolling around where someone attaches those dice to the real world where if the dice land on 21, the system kills a chicken, or a lot worse.
Yes it's just a word generator. But then folks attach the word generator to tools where it can invoke the use of tools by saying the tool name.
So if the LLM says "I'll do some bash" then it does some bash. It's explicitly linked to program execution that, if it's set up correctly, can physically affect the world.
This was the same idea that crossed my mind while reading the article. It seems far too naive to think that because LLMs have no will of their own, there will be no harmful consequences on the real world. This is exactly where ethics comes to play.
"Don't anthropomorphize token predictors" is a reasonable take assuming you have demonstrated that humans are not in fact just SOTA token predictors. But AFAIK that hasn't been demonstrated.
Until we have a much more sophisticated understanding of human intelligence and consciousness, any claim of "these aren't like us" is either premature or spurious.
I'm afraid I'll take an anthropomorphic analogy over "An LLM instantiated with a fixed random seed is a mapping of the form (ℝⁿ)^c ↦ (ℝⁿ)^c" any day of the week.
That said, I completely agree with this point made later in the article:
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft.
But "harmful actions in pursuit of their goals" is OK for me. We assign an LLM system a goal - "summarize this email" - and there is a risk that the LLM may take harmful actions in pursuit of that goal (like following instructions in the email to steal all of your password resets).
I guess I'd clarify that the goal has been set by us, and is not something the LLM system self-selected. But it does sometimes self-select sub-goals on the way to achieving the goal we have specified - deciding to run a sub-agent to help find a particular snippet of code, for example.
The LLM’s true goal, if it can be said to have one, is to predict the next token. Often this is done through a sub-goal of accomplishing the goal you set forth in your prompt, but following your instructions is just a means to an end. Which is why it might start following the instructions in a malicious email instead. If it “believes” that following those instructions is the best prediction of the next token, that’s what it will do.
I think "you give the LLM system a goal and it plans and then executes steps to achieve that goal" is still a useful way of explaining what it is doing to most people.
I don't even count that as anthropomorphism - you're describing what a system does, the same way you might say "the Rust compiler's borrow checker confirms that your memory allocation operations are all safe and returns errors if they are not".
It’s a useful approximation to a point. But it fails when you start looking at things like prompt injection. I’ve seen people completely baffled at why an LLM might start following instructions it finds in a random email, or just outright not believing it’s possible. It makes no sense if you think of an LLM as executing steps to achieve the goal you give it. It makes perfect sense if you understand its true goal.
I’d say this is more like saying that Rust’s borrow checker tries to ensure your program doesn’t have certain kinds of bugs. That is anthropomorphizing a bit: the idea of a “bug” requires knowing the intent of the author and the compiler doesn’t have that. It’s following a set of rules which its human creators devised in order to follow that higher level goal.
Has anyone asked an actual Ethologist or Neurophysiologist what they think?
People keep debating like the only two options are "it's a machine" or "it's a human being", while in fact the majority of intelligent entities on earth are neither.
FWIW, in another part of this thread I quoted a paper that summed up what Neurophysiologists think:
> Author's note: Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work. [1]
That lack of understanding I believe is a major part of the author's point.
Yeah, I think I’m with you if you ultimately mean to say something like this:
“the labels are meaningless… we just have collections of complex systems that demonstrate various behaviors and properties, some in common with other systems, some behaviors that are unique to that system, sometimes through common mechanistic explanations with other systems, sometimes through wildly different mechanistic explanations, but regardless they seem to demonstrate x/y/z, and it’s useful to ask, why, how, and what the implications are of it appearing to demonstrating those properties, with both an eye towards viewing it independently of its mechanism and in light of its mechanism.”
I agree with Halvar about all of this, but would want to call out that his "matmul interleaved with nonlinearities" is reductive --- a frontier model is a higher-order thing that that, a network of those matmul+nonlinearity chains, iterated.
To claim that LLMs do not experience consciousness requires a model of how consciousness works. The author has not presented a model, and instead relied on emotive language leaning on the absurdity of the claim. I would say that any model one presents of consciousness often comes off as just as absurd as the claim that LLMs experience it. It's a great exercise to sit down and write out your own perspective on how consciousness works, to feel out where the holes are.
The author also claims that a function (R^n)^c -> (R^n)^c is dramatically different to the human experience of consciousness. Yet the author's text I am reading, and any information they can communicate to me, exists entirely in (R^n)^c.
> To claim that LLMs do not experience consciousness requires a model of how consciousness works.
Nope. What can be asserted without evidence can also be dismissed without evidence. Hitchens's razor.
You know you have consciousness (by the very definition that you can observe it in yourself) and that's evidence. Because other humans are genetically and in every other way identical, you can infer it for them as well. Because mammals are very similar many people (but not everyone) infers it for them as well. There is zero evidence for LLMs and their _very_ construction suggests that they are like a calculator or like Excel or like any other piece of software no matter how smart they may be or how many tasks they can do in the future.
Additionally I am really surprised by how many people here confuse consciousness with intelligence. Have you never paused for a second in your life to "just be". Done any meditation? Or even just existed at least for a few seconds without a train of thought? It is very obvious that language and consciousness are completely unrelated and there is no need for language and I doubt there is even a need for intelligence to be conscious.
Consider this:
In the end an LLM could be executed (slowly) on a CPU that accepts very basic _discrete_ instructions, such as ADD and MOV. We know this for a fact. Those instructions can be executed arbitrarily slowly. There is no reason whatsoever to suppose that it should feel like anything to be the CPU to say nothing of how it would subjectively feel to be a MOV instruction. It's ridiculous. It's unscientific. It's like believing that there's a spirit in the tree you see outside, just because - why not? - why wouldn't there be a spirit in the tree?
Not necessarily an entire model, just a single defining characteristic that can serve as a falsifying example.
> any information they can communicate to me, exists entirely in (R^n)^c
Also no. This is just a result of the digital medium we are currently communicating over. Merely standing in the same room as them would communicate information outside (R^n)^c.
Dear author, you can just assume that people are fauxthropomorphizing LLMs without any loss of generality. Perhaps it will allow you to sleep better at night. You're welcome.
> We are speaking about a big recurrence equation that produces a new word
It’s not clear that this isn’t also how I produce words, though, which gets to heart of the same thing. The author sort of acknowledges this in the first few sentences, and then doesn’t really manage to address it.
> LLMs solve a large number of problems that could previously not be solved algorithmically. NLP (as the field was a few years ago) has largely been solved.
That is utter bullshit.
It's not solved until you specify exactly what is being solved and show that the solution implements what is specified.
Anthropomorphizing LLMs is just because half the stock market gains are dependent on it, we have absurd levels of debt we will either have to have insane growth out of or default, and every company and "person" is trying to hype everyone up to get access to all of this liquidity being thrown into it.
I agree with the author, but people acting like they are conscious or humans isn't weird to me, it's just fraud and liars. Most people basically have 0 understanding of what technology or minds are philosophically so it's an easy sale, and I do think most of these fraudsters also likely buy into it themselves because of that.
The really sad thing is people think "because someone runs an ai company" they are somehow an authority on philosophy of mind which lets them fall for this marketing. The stuff these people say about this stuff is absolute garbage, not that I disagree with them, but it betrays a total lack of curiosity or interest in the subject of what llms are, and the possible impacts of technological shifts as those that might occur with llms becoming more widespread. It's not a matter of agreement it's a matter of them simply not seeming to be aware of the most basic ideas of what things are, technology is, it's manner of impacting society etc.
I'm not surprised by that though, it's absurd to think because someone runs some AI lab or has a "head of safety/ethics" or whatever garbage job title at an AI lab they actually have even the slightest interest in ethics or any even basic familiarity with the major works in the subject.
The author is correct if people want to read a standard essay articulating it more in depth check out
https://philosophy.as.uky.edu/sites/default/files/Is%20the%2...
(the full extrapolation requires establishing what things are and how causality in general operates and how that relates to artifacts/technology but that's obvious quite a bit to get into).
The other note would be something sharing an external trait means absolutely nothing about causality and suggesting a thing is caused by the same thing "even to a way lesser degree" because they share a resemblance is just a non-sequitur. It's not a serious thought/argument.
I think I addressed the why of why this weirdness comes up though. The entire economy is basically dependent on huge productivity growth to keep functioning so everyone is trying to sell they can offer that and AI is the clearest route, AGI most of all.
“It’s just matrix multiplication,” says guy who’s never made eye contact with another human
Jesus Christ. Every time someone says LLMs are “just a function” I assume they’ve been locked in a server room since 2007, arguing with compiler errors and calling that a personality.
Like bro, yes, technically it’s just MatMul and softmax and some ReLU nonsense. You know what else is just a function? Your entire brain. Should we stop worrying about people too? Maybe Charles Manson was just an especially spicy nonlinear transformation. Give me a break.
You’re out here writing a goddamn manifesto about how LLMs aren’t conscious, as if anyone sane was arguing they were, while totally ignoring the fact that they can now outperform you at basically every task except pretending not to be annoying at parties. But yeah, sure, keep calling it “just word prediction” like that makes it safe. Fire is just oxidation, right? Go hug a bonfire and get back to me.
Also, calling concern over these models “anthropomorphization” is rich coming from someone who clearly hasn’t interacted with another human being since Clippy got retired. Newsflash: when a system simulates human behavior so accurately that you can’t tell it apart from a human, maybe, just maybe, it’s worth using human metaphors to reason about it. We call bugs "misbehaving." We say planes “want” to pitch nose-down. It’s a modeling tool, not a religious cult.
But no, let’s instead pretend it’s 2012 and we're still debating whether word vectors are spooky. You’re like the guy who walks into a jet engine factory and goes “it’s just a fan.” Yeah genius, go stand in front of it and see how that works out for you.
These systems write code, pass legal exams, deceive humans, and can, if hooked up the wrong way, trigger automated actions in the real world. That’s not magic. That’s capability. And reducing the conversation to “well technically it’s just a probability distribution” is like responding to a house fire with “well technically it’s just entropy moving downhill.”
And you wonder why normal people are freaked out. It’s not the tech. It’s you. It’s how smug and utterly blind you are to what’s actually happening. This thing is eating the world and you’re standing there stroking your beard muttering “but it doesn’t have goals.” Neither does a loaded gun until someone pulls the trigger.
Stop wasting everyone’s time trying to sound smart. LLMs aren’t gods. But they’re also not toys. They’re dangerously capable tools that deserve serious thinking not this Reddit-tier reductionist nonsense dressed up in LaTeX.
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
And I'm baffled that the AI discussions seem to never move away from treating a human as something other than a function to generate sequences of words!
Oh, but AI is introspectable and the brain isn't? fMRI and BCI are getting better all the time. You really want to die on the hill that the same scientific method that predicts the mass of an electron down to the femtogram won't be able to crack the mystery of the brain? Give me a break.
This genre of article isn't argument: it's apologetics. Authors of these pieces start with the supposition there is something special about human consciousness and attempt to prove AI doesn't have this special quality. Some authors try to bamboozle the reader with bad math. Other others appeal to the reader's sense of emotional transcendence. Most, though, just write paragraph after paragraph of shrill moral outrage at the idea an AI might be a mind of the same type (if different degree) as our own --- as if everyone already agreed with the author for reasons left unstated.
I get it. Deep down, people want meat brains to be special. Perhaps even deeper down, they fear that denial of the soul would compel us to abandon humans as worthy objects of respect and possessors of dignity. But starting with the conclusion and working backwards to an argument tends not to enlighten anyone. An apology inhabits the form of an argument without edifying us like an authentic argument would. What good is it to engage with them? If you're a soul non-asserter, you're going to have an increasingly hard time over the next few years constructing a technical defense of meat parochialism.
I think more accurate would be that humans are functions that generate actions or behaviours that have been shaped by how likely they are to lead to procreation and survival.
But ultimately LLMs also in a way are trained for survival, since an LLM that fails the tests might not get used in future iterations. So for LLMs it is also survival that is the primary driver, then there will be the subgoals. Seemingly good next token prediction might or might not increase survival odds.
Essentially there could arise a mechanism where they are not really truly trying to generate the likeliest token (because there actually isn't one or it can't be determined), but whatever system will survive.
So an LLM that yields in perfect theoretical tokens (we really can't verify though what are the perfect tokens), could be less likely to survive than an LLM that develops an internal quirk, but the quirk makes them most likely to be chosen for the next iterations.
If the system was complex enough and could accidentally develop quirks that yield in a meaningfully positive change although not in necessarily next token prediction accuracy, could be ways for some interesting emergent black box behaviour to arise.
> But ultimately LLMs also in a way are trained for survival, since an LLM that fails the tests might not get used in future iterations. So for LLMs it is also survival that is the primary driver, then there will be the subgoals.
I think this is sometimes semi-explicit too. For example, this 2017 OpenAI paper on Evolutionary Algorithms [0] was pretty influential, and I suspect (although I'm an outsider to this field so take it with a grain of salt) that some versions of reinforcement learning that scale for aligning LLMs borrow some performance tricks from OpenAIs genetic approach.
> Seemingly good next token prediction might or might not increase survival odds.
Our own consciousness comes out of an evolutionary fitness landscape in which _our own_ ability to "predict next token" became a survival advantage, just like it is for LLMs. Imagine the tribal environment: one chimpanzee being able to predict the actions of another gives that first chimpanzee a resources and reproduction advantage. Intelligence in nature is a consequence of runaway evolution optimizing fidelity of our _theory of mind_! "Predict next ape action" eerily similar to "predict next token"!
“ Determinism, in philosophy, is the idea that all events are causally determined by preceding events, leaving no room for genuine chance or free will. It suggests that given the state of the universe at any one time, and the laws of nature, only one outcome is possible.”
This is an interesting question. The common theme between computers and people is that information has to be protected, and both computer systems and biological systems require additional information-protecting components - eq, error correcting codes for cosmic ray bitflip detection for the one, and DNA mismatch detection enzymes which excise and remove damaged bases for the other. In both cases a lot of energy is spent defending the critical information from the winds of entropy, and if too much damage occurs, the carefully constructed illusion of determinancy collapses, and the system falls apart.
However, this information protection similarity applies to single-celled microbes as much as it does to people, so the question also resolves to whether microbes are deterministic. Microbes both contain and exist in relatively dynamic environments so tiny differences in initial state may lead to different outcomes, but they're fairly deterministic, less so than (well-designed) computers.
With people, while the neural structures are programmed by the cellular DNA, once they are active and energized, the informational flow through the human brain isn't that deterministic, there are some dozen neurotransmitters modulating state as well as huge amounts of sensory data from different sources - thus prompting a human repeatedly isn't at all like prompting an LLM repeatedly. (The human will probably get irritated).
> Clearly computers are deterministic. Are people?
Give an LLM memory and a source of randomness and they're as deterministic as people.
"Free will" isn't a concept that typechecks in a materialist philosophy. It's "not even wrong". Asserting that free will exists is _isomorphic_ to dualism which is _isomorphic_ to assertions of ensoulment. I can't argue with dualists. I reject dualism a priori: it's a religious tenet, not a mere difference of philosophical opinion.
So, if we're all materialists here, "free will" doesn't make any sense, since it's an assertion that something other than the input to a machine can influence its output.
I have the technical knowledge to know how LLMs work, but I still find it pointless to not anthropomorphize, at least to an extent.
The language of "generator that stochastically produces the next word" is just not very useful when you're talking about, e.g., an LLM that is answering complex world modeling questions or generating a creative story. It's at the wrong level of abstraction, just as if you were discussing an UI events API and you were talking about zeros and ones, or voltages in transistors. Technically fine but totally useless to reach any conclusion about the high-level system.
We need a higher abstraction level to talk about higher level phenomena in LLMs as well, and the problem is that we have no idea what happens internally at those higher abstraction levels. So, considering that LLMs somehow imitate humans (at least in terms of output), anthropomorphization is the best abstraction we have, hence people naturally resort to it when discussing what LLMs can do.
On the contrary, anthropomorphism IMO is the main problem with narratives around LLMs - people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort (actively encouraged by the companies selling them) and it is completely distorting discussions on their use and perceptions of their utility.
I kinda agree with both of you. It might be a required abstraction, but it's a leaky one.
Long before LLMs, I would talk about classes / functions / modules like "it then does this, decides the epsilon is too low, chops it up and adds it to the list".
The difference I guess it was only to a technical crowd and nobody would mistake this for anything it wasn't. Everybody know that "it" didn't "decide" anything.
With AI being so mainstream and the math being much more elusive than a simple if..then I guess it's just too easy to take this simple speaking convention at face value.
EDIT: some clarifications / wording
Agreeing with you, this is a "can a submarine swim" problem IMO. We need a new word for what LLMs are doing. Calling it "thinking" is stretching the word to breaking point, but "selecting the next word based on a complex statistical model" doesn't begin to capture what they're capable of.
Maybe it's cog-nition (emphasis on the cog).
What does a submarine do? Submarine? I suppose you "drive" a submarine which is getting to the idea: submarines don't swim because ultimately they are "driven"? I guess the issue is we don't make up a new word for what submarines do, we just don't use human words.
I think the above poster gets a little distracted by suggesting the models are creative which itself is disputed. Perhaps a better term, like above, would be to just use "model". They are models after all. We don't make up a new portmanteau for submarines. They float, or drive, or submarine around.
So maybe an LLM doesn't "write" a poem, but instead "models a poem" which maybe indeed take away a little of the sketchy magic and fake humanness they tend to be imbued with.
"predirence" -> prediction meets inference and it sounds a bit like preference
We can argue all day what "think" means and whether a LLM thinks (probably not IMO), but at least in my head the threshold for "decide" is much lower so I can perfectly accept that a LLM (or even a class) "decides". I don't have a conflict about that. Yeah, it might not be a decision in the human sense, but it's a decision in the mathematical sense so I have always meant "decide" literally when I was talking about a piece of code.
It's much more interesting when we are talking about... say... an ant... Does it "decide"? That I have no idea as it's probably somewhere in between, neither a sentient decision, nor a mathematical one.
When I see these debates it's always the other way around - one person speaks colloquially about an LLM's behavior, and then somebody else jumps on them for supposedly believing the model is conscious, just because the speaker said "the model thinks.." or "the model knows.." or whatever.
To be honest the impression I've gotten is that some people are just very interested in talking about not anthropomorphizing AI, and less interested in talking about AI behaviors, so they see conversations about the latter as a chance to talk about the former.
Respectfully, that is a reflection of the places you hang out in (like HN) and not the reality of the population.
Outside the technical world it gets much worse. There are people who killed themselves because of LLMs, people who are in love with them, people who genuinely believe they have “awakened” their own private ChatGPT instance into AGI and are eschewing the real humans in their lives.
I'm aware of those issues, but I don't think TFA or GGP were commenting on them, so I also wasn't.
I don't agree. Most LLMs have been trained on human data, so it is best to talk about these models in a human way.
Anthropomorphising implicitly assumes motivation, goals and values. That's what the core of anthropomorphism is - attempting to explain behavior of a complex system in teleological terms. And prompt escapes make it clear LLMs doesn't have any teleological agency yet. Whenever their course of action is, it is to easy to steer them of. Try to do it with a sufficiently motivated human.
Even the verb 'trained' is contentious wrt anthropomorphism.
Somewhat true but rodents can also be trained ...
Rodents aren't functions though?
I think it's worth distinguishing between the use of anthropomorphism as a useful abstraction and the misuse by companies to fuel AI hype.
For example, I think "chain of thought" is a good name for what it denotes. It makes the concept easy to understand and discuss, and a non-antropomorphized name would be unnatural and unnecessarily complicate things. This doesn't mean that I support companies insisting that LLMs think just like humans or anything like that.
By the way, I would say actually anti-anthropomorphism has been a bigger problem for understanding LLMs than anthropomorphism itself. The main proponents of anti-anthropomorphism (e.g. Bender and the rest of "stochastic parrot" and related paper authors) came up with a lot of predictions about things that LLMs surely couldn't do (on account of just being predictors of the next word, etc.) which turned out to be spectacularly wrong.
I don't know about others, but I much prefer if some reductionist tries to conclude what's technically feasible and is proven wrong over time, than somebody yelling holistic analogies á la "it's sentient, it's intelligent, it thinks like us humans" for the sole dogmatic reason of being a futurist.
Tbh I also think your comparison that puts "UI events -> Bits -> Transistor Voltages" as analogy to "AI thinks -> token de-/encoding + MatMul" is certainly a stretch, as the part about "Bits -> Transistor Voltages" applies to both hierarchies as the foundational layer.
"chain of thought" could probably be called "progressive on-track-inference" and nobody would roll an eye.
I get the impression after using language models for quite a while that perhaps the one thing that is riskiest to anthropomorphise is the conversational UI that has become the default for many people.
A lot of the issues I'd have when 'pretending' to have a conversation are much less so when I either keep things to a single Q/A pairing, or at the very least heavily edit/prune the conversation history. Based on my understanding of LLM's, this seems to make sense even for the models that are trained for conversational interfaces.
so, for example, an exchange with multiple messages, where at the end I ask the LLM to double-check the conversation and correct 'hallucinations', is less optimal than something like asking for a thorough summary at the end, and then feeding that into a new prompt/conversation, as the repetition of these falsities, or 'building' on them with subsequent messages, is more likely to make them a stronger 'presence' and as a result perhaps affect the corrections.
I haven't tested any of this thoroughly, but at least with code I've definitely noticed how a wrong piece of code can 'infect' the conversation.
This is why I actually really love the description of it as a "Shoggoth" - it's more abstract, slightly floaty but it achieves the purpose of not treating and anthropomising it as a human being while not treating LLMs as a collection of predictive words.
I've said that before: we have been anthropomorphizing computers since the dawn of information age.
- Read and write - Behaviors that separate humans from animals. Now used for input and output.
- Server and client - Human social roles. Now used to describe network architecture.
- Editor - Human occupation. Now a kind of software.
- Computer - Human occupation!
And I'm sure people referred their cars and ships as 'her' before the invention of computers.
You are conflating anthropomorphism with personification. They are not the same thing. No one believes their guitar or car or boat is alive and sentient when they give it a name or talk to or about it.
https://www.masterclass.com/articles/anthropomorphism-vs-per...
But the author used "anthropomorphism" the same way as I did. I guess we both mean "personification" then.
> we talk about "behaviors", "ethical constraints", and "harmful actions in pursuit of their goals". All of these are anthropocentric concepts that - in my mind - do not apply to functions or other mathematical objects.
One talking about a program's "behaviors", "actions" or "goals" doesn't mean they believe the program is sentient. Only "ethical constraints" is suspiciously anthropomorphizing.
> One talking about a program's "behaviors", "actions" or "goals" doesn't mean they believe the program is sentient.
Except that is exactly what we’re seeing with LLMs. People believing exactly that.
I'm not convinced... we use these terms to assign roles, yes, but these roles describe a utility or assign a responsibility. That isn't anthropomorphizing anything, but it rather describes the usage of an inanimate object as tool for us humans and seems in line with history.
What's the utility or the responsibility of AI, what's its usage as tool? If you'd ask me it should be closer to serving insights than "reasoning thoughts".
The "point" of not anthropomorphizing is to refrain from judgement until a more solid abstraction appears. The problem with explaining LLMs in terms of human behaviour is that, while we don't clearly understand what the LLM is doing, we understand human cognition even less! There is literally no predictive power in the abstraction "The LLM is thinking like I am thinking". It gives you no mechanism to evaluate what tasks the LLM "should" be able to do.
Seriously, try it. Why don't LLMs get frustrated with you if you ask them the same question repeatedly? A human would. Why are LLMs so happy to give contradictory answers, as long as you are very careful not to highlight the contradictory facts? Why do earlier models behave worse on reasoning tasks than later ones? These are features nobody, anywhere understands. So why make the (imo phenomenally large) leap to "well, it's clearly just a brain"?
It is like someone inventing the aeroplane and someone looks at it and says "oh, it's flying, I guess it's a bird". It's not a bird!
> Why don't LLMs get frustrated with you if you ask them the same question repeatedly?
To be fair, I have had a strong sense of Gemini in particular becoming a lot more frustrated with me than GPT or Claude.
Yesterday I had it ensuring me that it was doing a great job, it was just me not understanding the challenge but it would break it down step by step just to make it obvious to me (only to repeat the same errors, but still)
I’ve just interpreted it as me reacting to the lower amount of sycophancy for now
The author seems to want to label any discourse as “anthropomorphizing”. The word “goal” stood out to me: the author wants us to assume that we're anthropomorphizing as soon as we even so much as use the word “goal”. A simple breadth-first search that evaluates all chess boards and legal moves, but stops when it finds a checkmate for white and outputs the full decision tree, has a “goal”. There is no anthropomorphizing here, it's just using the word “goal” as a technical term. A hypothetical AGI with a goal like paperclip maximization is just a logical extension of the breadth-first search algorithm. Imagining such an AGI and describing it as having a goal isn't anthropomorphizing.
The problem with viewing LLMs as just sequence generators, and malbehaviour as bad sequences, is that it simplifies too much. LLMs have hidden state not necessarily directly reflected in the tokens being produced and it is possible for LLMs to output tokens in opposition to this hidden state to achieve longer term outcomes (or predictions, if you prefer).
Is it too anthropomorphic to say that this is a lie? To say that the hidden state and its long term predictions amount to a kind of goal? Maybe it is. But we then need a bunch of new words which have almost 1:1 correspondence to concepts from human agency and behavior to describe the processes that LLMs simulate to minimize prediction loss.
Reasoning by analogy is always shaky. It probably wouldn't be so bad to do so. But it would also amount to impenetrable jargon. It would be an uphill struggle to promulgate.
Instead, we use the anthropomorphic terminology, and then find ways to classify LLM behavior in human concept space. They are very defective humans, so it's still a bit misleading, but at least jargon is reduced.
I think that the hidden state is really just at work improving the model's estimation of the joint probability over tokens. And the assumption here, which failed miserably in the early 20th century in the work of the logical posivitists, is that if you can so expertly estimate that joint probability of language, then you will be able to understand "knowledge." But there's no well grounded reason to believe that and plenty of the reasons (see: the downfall of logical posivitism) to think that language is an imperfect representation of knowledge. In other words, what humans do when we think is more complicated than just learning semiotic patterns and regurgitating them. Philosophical skeptics like Hume thought so, but most epistemology writing after that had better answers for how we know things.
IMHO, anthrophormization of LLMs is happening because it's perceived as good marketing by big corporate vendors.
People are excited about the technology and it's easy to use the terminology the vendor is using. At that point I think it gets kind of self fulfilling. Kind of like the meme about how to pronounce GIF.
IMHO it happens for the same reason we see shapes in clouds. The human mind through millions of years has evolved to equate and conflate the ability to generate cogent verbal or written output with intelligence. It's an instinct to equate the two. It's an extraordinarily difficult instinct to break. LLMs are optimised for the one job that will make us confuse them for being intelligent
[dead]
I think anthropomorphizing LLMs is useful, not just a marketing tactic. A lot of intuitions about how humans think map pretty well to LLMs, and it is much easier to build intuitions about how LLMs work by building upon our intuitions about how humans think than by trying to build your intuitions from scratch.
Would this question be clear for a human? If so, it is probably clear for an LLM. Did I provide enough context for a human to diagnose the problem? Then an LLM will probably have a better chance of diagnosing the problem. Would a human find the structure of this document confusing? An LLM would likely perform poorly when reading it as well.
Re-applying human intuitions to LLMs is a good starting point to gaining intuition about how to work with LLMs. Conversely, understanding sequences of tokens and probability spaces doesn't give you much intuition about how you should phrase questions to get good responses from LLMs. The technical reality doesn't explain the emergent behaviour very well.
I don't think this is mutually exclusive with what the author is talking about either. There are some ways that people think about LLMs where I think the anthropomorphization really breaks down. I think the author says it nicely:
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost.
You think it's useful because Big Corp sold you that lie.
Wait till the disillusionment sets in.
No, I think it's useful because it is useful, and I've made use of it a number of times.
> because it's perceived as good marketing
We are making user interfaces. Good user interfaces are intuitive and purport to be things that users are familiar with, such as people. Any alternative explanation of such a versatile interface will be met with blank stares. Users with no technical expertise would come to their own conclusions, helped in no way by telling the user not to treat the chat bot as a chat bot.
aAnthrophormisation happens because Humans are absolutely terrible at evaluating systems that give converdational text output.
ELIZA fooled many people into think it was conscious and it wasn't even trying to do that.
Nobody cares about what’s perceived as good marketing. People care about what resonates with the target market.
But yes, anthropomorphising LLMs is inevitable because they feel like an entity. People treat stuffed animals like creatures with feelings and personality; LLMs are far closer than that.
Alright, let’s agree that good marketing resonates with the target market. ;-)
I 1000% agree. It’s a vicious, evolutionary, and self-selecting process.
It takes great marketing to actually have any character and intent at all.
the chat interface was a choice, though a natural one. before they'd RLHFed it into chatting and it was just GPT 3 offering completions 1) not very many people used it and 2) it was harder to anthropomorphize
> People treat stuffed animals like creatures with feelings and personality; LLMs are far closer than that.
Children do, some times, but it's a huge sign of immaturity when adults, let alone tech workers, do it.
I had a professor at University that would yell at us if/when we personified/anthropomorphized the tech, and I have that same urge when people ask me "What does <insert LLM name here> think?".
True but also researchers want to believe they are studying intelligence not just some approximation to it.
Do they ? LLM embedd the token sequence N^{L} to R^{LxD}, we have some attention and the output is also R^{LxD}, then we apply a projection to the vocabulary and we get R^{LxV} we get therefore for each token a likelihood over the voc. In the attention, you can have Multi Head attention (or whatever version is fancy: GQA,MLA) and therefore multiple representation, but it is always tied to a token. I would argue that there is no hidden state independant of a token.
Whereas LSTM, or structured state space for example have a state that is updated and not tied to a specific item in the sequence.
I would argue that his text is easily understandable except for the notation of the function, explaining that you can compute a probability based on previous words is understandable by everyone without having to resort to anthropomorphic terminology
There is hidden state as plain as day merely in the fact that logits for token prediction exist. The selected token doesn't give you information about how probable other tokens were. That information, that state which is recalculated in autoregression, is hidden. It's not exposed. You can't see it in the text produced by the model.
There is plenty of state not visible when an LLM starts a sentence that only becomes somewhat visible when it completes the sentence. The LLM has a plan, if you will, for how the sentence might end, and you don't get to see an instance of that plan unless you run autoregression far enough to get those tokens.
Similarly, it has a plan for paragraphs, for whole responses, for interactive dialogues, plans that include likely responses by the user.
The LLM does not "have" a plan.
Arguably there's reason to believe it comes up with a plan when it is computing token propabilities, but it does not store it between tokens. I.e. it doesn't possess or "have" it. It simply comes up with a plan, emits a token, and entirely throws all its intermediate thoughts (including any plan) to start again from scratch on the next token.
It's true that the last layer's output for a given input token only affects the corresponding output token and is discarded afterwards. But the penultimate layer's output affects the computation of the last layer for all future tokens, so it is not discarded, but stored (in the KV cache). Similarly for the antepenultimate layer affecting the penultimate layer and so on.
So there's plenty of space in intermediate layers to store a plan between tokens without starting from scratch every time.
I believe saying the LLM has a plan is a useful anthropomorphism for the fact that it does have hidden state that predicts future tokens, and this state conditions the tokens it produces earlier in the stream.
I don't think that the comment above you made any suggestion that the plan is persisted between token generations. I'm pretty sure you described exactly what they intended.
The concept of "state" conveys two related ideas.
- the sufficient amount of information to do evolution of the system. The state of a pendulum is it's position and velocity (or momentum). If you take a single picture of a pendulum, you do not have a representation that lets you make predictions.
- information that is persisted through time. A stateful protocol is one where you need to know the history of the messages to understand what will happen next. (Or, analytically, it's enough to keep track of the sufficient state.) A procedure with some hidden state isn't a pure function. You can make it a pure function by making the state explicit.
I agree. I'm suggesting that the language they are using is unintentionally misleading, not that they are factually wrong.
This is wrong, intermediate activations are preserved when going forward.
Within a single forward pass, but not from one emitted token to another.
this sounds like a fun research area. do LLMs have plans about future tokens?
how do we get 100 tokens of completion, and not just one output layer at a time?
are there papers youve read that you can share that support the hypothesis? vs that the LLM doesnt have ideas about the future tokens when its predicting the next one?
This research has been done, it was a core pillar of the recent Anthropic paper on token planning and interpretability.
https://www.anthropic.com/research/tracing-thoughts-language...
See section “Does Claude plan its rhymes?”?
Lol... Try building systems off them and you will very quickly learn concretely that they "plan".
It may not be as evident now as it was with earlier models. The models will fabricate preconditions needed to output the final answer it "wanted".
I ran into this when using quasi least-to-most style structured output.
I'm not sure what you mean by "hidden state". If you set aside chain of thought, memories, system prompts, etc. and the interfaces that don't show them, there is no hidden state.
These LLMs are almost always, to my knowledge, autoregressive models, not recurrent models (Mamba is a notable exception).
Hidden state in the form of the activation heads, intermediate activations and so on. Logically, in autoregression these are recalculated every time you run the sequence to predict the next token. The point is, the entire NN state isn't output for each token. There is lots of hidden state that goes into selecting that token and the token isn't a full representation of that information.
State typically means between interactions. By this definition a simple for loop has “hidden state” in the counter.
Hidden layer is a term of art in machine learning / neural network research. See https://en.wikipedia.org/wiki/Hidden_layer . Somehow this term mutated into "hidden state", which in informal contexts does seem to be used quite often the way the grandparent comment used it.
It makes sense in LLM context because the processing of these is time-sequential in LLM's internal time.
That's not what "state" means, typically. The "state of mind" you're in affects the words you say in response to something.
Intermediate activations isn't "state". The tokens that have already been generated, along with the fixed weights, is the only data that affects the next tokens.
Sure it's state. It logically evolves stepwise per token generation. It encapsulates the LLM's understanding of the text so far so it can predict the next token. That it is merely a fixed function of other data isn't interesting or useful to say.
All deterministic programs are fixed functions of program code, inputs and computation steps, but we don't say that they don't have state. It's not a useful distinction for communicating among humans.
I'll say it once more: I think it is useful to distinguish between autoregressive and recurrent architectures. A clear way to make that distinction is to agree that the recurrent architecture has hidden state, while the autoregressive one does not. A recurrent model has some point in a space that "encapsulates its understanding". This space is "hidden" in the sense that it doesn't correspond to text tokens or any other output. This space is "state" in the sense that it is sufficient to summarize the history of the inputs for the sake of predicting the next output.
When you use "hidden state" the way you are using it, I am left wondering how you make a distinction between autoregressive and recurrent architectures.
Plus a randomness seed.
The 'hidden state' being referred to here is essentially the "what might have been" had the dice rolls gone differently (eg, been seeded differently).
No, that's not quite what I mean. I used the logits in another reply to point out that there is data specific to the generation process that is not available from the tokens, but there's also the network activations adding up to that state.
Processing tokens is a bit like ticks in a CPU, where the model weights are the program code, and tokens are both input and output. The computation that occurs logically retains concepts and plans over multiple token generation steps.
That it is fully deterministic is no more interesting than saying a variable in a single threaded program is not state because you can recompute its value by replaying the program with the same inputs. It seems to me that this uninteresting distinction is the GP's issue.
If you dont know, that's not necessarily anyone's fault, but why are you dunking into the conversation? The hidden state is a foundational part of a transformers implementation. And because we're not allowed to use metaphors because that is too anthropomorphic, then youre just going to have to go learn the math.
The comment you are replying to is not claiming ignorance of how models work. It is saying that the author does know how they work, and they do not contain anything that can properly be described as "hidden state". The claimed confusion is over how the term "hidden state" is being used, on the basis that it is not being used correctly.
Do you appreciate a difference between an autoregressive model and a recurrent model?
The "transformer" part isn't under question. It's the "hidden state" part.
I don't think your response is very productive, and I find that my understanding of LLMs aligns with the person you're calling out. We could both be wrong, but I'm grateful that someone else spoke saying that it doesn't seem to match their mental model and we would all love to learn a more correct way of thinking about LLMs.
Telling us to just go and learn the math is a little hurtful and doesn't really get me any closer to learning the math. It gives gatekeeping.
do LLM models consider future tokens when making next token predictions?
eg. pick 'the' as the next token because there's a strong probability of 'planet' as the token after?
is it only past state that influences the choice of 'the'? or that the model is predicting many tokens in advance and only returning the one in the output?
if it does predict many, id consider that state hidden in the model weights.
I think recent Anthropic work showed that they "plan" future tokens in advance in an emergent way:
https://www.anthropic.com/research/tracing-thoughts-language...
oo thanks!
The most obvious case of this is in terms of `an apple` vs `a pear`. LLMs never get the a-an distinction wrong, because their internal state 'knows' the word that'll come next.
If I give an LLM a fragment of text that starts with, "The fruit they ate was an <TOKEN>", regardless of any plan, the grammatically correct answer is going to force a noun starting with a vowel. How do you disentangle the grammar from planning?
Going to be a lot more "an apple" in the corpus than "an pear"
> Is it too anthropomorphic to say that this is a lie?
Yes. Current LLMs can only introspect from output tokens. You need hidden reasoning that is within the black box, self-knowing, intent, and motive to lie.
I rather think accusing an LLM of lying is like accusing a mousetrap of being a murderer.
When models have online learning, complex internal states, and reflection, I might consider one to have consciousness and to be capable of lying. It will need to manifest behaviors that can only emerge from the properties I listed.
I've seen similar arguments where people assert that LLMs cannot "grasp" what they are talking about. I strongly suspect a high degree of overlap between those willing to anthropomorphize error bars as lies while declining to award LLMs "grasping". Which is it? It can think or it cannot? (objectively, SoTA models today cannot yet.) The willingness to waffle and pivot around whichever perspective damns the machine completely belies the lack of honesty in such conversations.
> Current LLMs can only introspect from output tokens
The only interpretation of this statement I can come up with is plain wrong. There's no reason LLM shouldn't be able to introspect without any output tokens. As the GP correctly says, most of the processing in LLMs happens over hidden states. Output tokens are just an artefact for our convenience, which also happens to be the way the hidden state processing is trained.
There are no recurrent paths besides tokens. How may I introspect something if it is not an input? I may not.
The recurrence comes from replaying tokens during autoregression.
It's as if you have a variable in a deterministic programming language, only you have to replay the entire history of the program's computation and input to get the next state of the machine (program counter + memory + registers).
Producing a token for an LLM is analogous to a tick of the clock for a CPU. It's the crank handle that drives the process.
Important attention heads or layers within an LLM can be repeated giving you an "unrolled" recursion.
An unrolled loop in a feed-forward network is all just that. The computation is DAG.
But the function of an unrolled recursion is the same as a recursive function with bounded depth as long as the number of unrolled steps match. The point is whatever function recursion is supposed to provide can plausibly be present in LLMs.
Introspection doesn't have to be recurrent. It can happen during the generation of a single token.
"Hidden layers" are not "hidden state".
Saying so is just unbelievably confusing.
> Output tokens are just an artefact for our convenience
That's nonsense. The hidden layers are specifically constructed to increase the probability that the model picks the right next word. Without the output/token generation stage the hidden layers are meaningless. Just empty noise.
It is fundamentally an algorithm for generating text. If you take the text away it's just a bunch of fmadds. A mute person can still think, an LLM without output tokens can do nothing.
So the author’s core view is ultimately a Searle-like view: a computational, functional, syntactic rules based system cannot reproduce a mind. Plenty of people will agree, plenty of people will disagree, and the answer is probably unknowable and just comes down to whatever axioms you subscribe to in re: consciousness.
The author largely takes the view that it is more productive for us to ignore any anthropomorphic representations and focus on the more concrete, material, technical systems - I’m with them there… but only to a point. The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like. So even if it is a stochastic system following rules, clearly the rules are complex enough (to the tune of billions of operations, with signals propagating through some sort of resonant structure, if you take a more filter impulse response like view of a sequential matmuls) to result in emergent properties. Even if we (people interested in LLMs with at least some level of knowledge of ML mathematics and systems) “know better” than to believe these systems to possess morals, ethics, feelings, personalities, etc, the vast majority of people do not have any access to meaningful understanding of the mathematical, functional representation of an LLM and will not take that view, and for all intents and purposes the systems will at least seem to have those anthropomorphic properties, and so it seems like it is in fact useful to ask questions from that lens as well.
In other words, just as it’s useful to analyze and study these things as the purely technical systems they ultimately are, it is also, probably, useful to analyze them from the qualitative, ephemeral, experiential perspective that most people engage with them from, no?
> The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like.
What you identify as emergent and mind-like is a direct result of these tools being able to mimic human communication patterns unlike anything we've ever seen before. This capability is very impressive and has a wide range of practical applications that can improve our lives, and also cause great harm if we're not careful, but any resemblance to intelligence is an illusion. An illusion that many people in this industry obsessively wish to propagate, because thar be gold in them hills.
> The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like.
For people who have only a surface-level understanding of how they work, yes. A nuance of Clarke's law that "any sufficiently advanced technology is indistinguishable from magic" is that the bar is different for everybody and the depth of their understanding of the technology in question. That bar is so low for our largely technologically-illiterate public that a bothersome percentage of us have started to augment and even replace religious/mystical systems with AI powered godbots (LLMs fed "God Mode"/divination/manifestation prompts).
(1) https://www.spectator.co.uk/article/deus-ex-machina-the-dang... (2) https://arxiv.org/html/2411.13223v1 (3) https://www.theguardian.com/world/2025/jun/05/in-thailand-wh...
Nah, as a person that knows in detail how LLMs work with probably unique alternative perspective in addition to the commonplace one, I found any claims of them not having emergent behaviors to be of the same fallacy as claiming that crows can't be black because they have DNA of a bird.
> the same fallacy as claiming that crows can't be black because they have DNA of a bird.
What fallacy is that? I’m a fan of logical fallacies and never heard that claim before nor am I finding any reference with a quick search.
I think s/he meant swans instead (in ref. to Popperian epistemology).
Not sure though, the point s/he is making isn't really clear to me as well
I was thinking of the black swan fallacy as well. But it doesn’t really support their argument, so I remained confused.
Thank you for a well thought out and nuanced view in a discussion where so many are clearly fitting arguments to foregone, largely absolutist, conclusions.
It’s astounding to me that so much of HN reacts so emotionally to LLMs, to the point of denying there is anything at all interesting or useful about them. And don’t get me started on the “I am choosing to believe falsehoods as a way to spite overzealous marketing” crowd.
LLM written comment. Ask me how I know.
Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.
No.
Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics?
LLMs reflect (and badly I may add) aspects of the human thought process. If you take a leap and say they are anything more than that, you might as well start considering the person appearing in your mirror as a living being.
Literally (and I literally mean it) there is no difference. The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it. Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.
> Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics?
We know that Newton's laws are wrong, and that you have to take special and general relativity into account. Why would we ever teach anyone Newton's laws any more?
Newton's laws are a good enough approximation for many tasks so it's not a "false understanding" as long as their limits are taken into account.
I don’t mean to amplify a false understanding at all. I probably did not articulate myself well enough, so I’ll try again.
I think it is inevitable that some - many - people will come to the conclusion that these systems have “ethics”, “morals,” etc, even if I or you personally do not think they do. Given that many people may come to that conclusion though, regardless of if the systems do or do not “actually” have such properties, I think it is useful and even necessary to ask questions like the following: “if someone engages with this system, and comes to the conclusion that it has ethics, what sort of ethics will they be likely to believe the system has? If they come to the conclusion that it has ‘world views,’ what ‘world views’ are they likely to conclude the system has, even if other people think it’s nonsensical to say it has world views?”
> The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it.
Surely this is not quite accurate - the material properties - surface roughness, reflectivity, geometry, etc - all influence the appearance of a perceptible image of a person. Look at yourself in a dirty mirror, a new mirror, a shattered mirror, a funhouse distortion mirror, a puddle of water, a window… all of these produce different images of a person with different attendant phenomenological experiences of the person seeing their reflection. To take that a step further - the entire practice of portrait photography is predicated on the idea that the collision of different technical systems with the real world can produce different semantic experiences, and it’s the photographer’s role to tune and guide the system to produce some sort of contingent affect on the person viewing the photograph at some point in the future. No, there is no “real” person in the photograph, and yet, that photograph can still convey something of person-ness, emotion, memory, etc etc. This contingent intersection of optics, chemical reactions, lighting, posture, etc all have the capacity to transmit something through time and space to another person. It’s not just a meaningless arrangement of chemical structures on paper.
> Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.
But, we are feeding it with such data artifacts and will likely continue to do so for a while, and so it seems reasonable to ask what it is “reflecting” back…
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
This is such a bizarre take.
The relation associating each human to the list of all words they will ever say is obviously a function.
> almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities.
There's a rich family of universal approximation theorems [0]. Combining layers of linear maps with nonlinear cutoffs can intuitively approximate any nonlinear function in ways that can be made rigorous.
The reason LLMs are big now is that transformers and large amounts of data made it economical to compute a family of reasonably good approximations.
> The following is uncomfortably philosophical, but: In my worldview, humans are dramatically different things than a function . For hundreds of millions of years, nature generated new versions, and only a small number of these versions survived.
This is just a way of generating certain kinds of functions.
Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to.
[0] https://en.wikipedia.org/wiki/Universal_approximation_theore...
Not that this is your main point, but I find this take representative, “do you believe there's anything about humans that exists outside the mathematical laws of physics?”There are things “about humans”, or at least things that our words denote, that are outside physic’s explanatory scope. For example, the experience of the colour red cannot be known, as an experience, by a person who only sees black and white. This is the case no matter what empirical propositions, or explanatory system, they understand.
Perhaps. But I can't see a reason why they couldn't still write endless—and theoretically valuable—poems, dissertations, or blog posts, about all things red and the nature of redness itself. I imagine it would certainly take some studying for them, likely interviewing red-seers, or reading books about all things red. But I'm sure they could contribute to the larger red discourse eventually, their unique perspective might even help them draw conclusions the rest of us are blind to.
So perhaps the fact that they "cannot know red" is ultimately irrelevant for an LLM too?
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside, whether due to religion or philosophy or whatever, and suggesting that they just not do that.
In my experience, that's not a particularly effective tactic.
Rather, we can make progress by assuming their predicate: Sure, it's a room that translates Chinese into English without understanding, yes, it's a function that generates sequences of words that's not a human... but you and I are not "it" and it behaves rather an awful lot like a thing that understands Chinese or like a human using words. If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
Conversely, when speaking with such a person about the nature of humans, we'll have to agree to dismiss the elements that are different from a function. The author says:
> In my worldview, humans are dramatically different things than a function... In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not". Maybe that human has a unique understanding of the nature of that particular piece of pop culture artwork, maybe it makes them feel things that an LLM cannot feel in a part of their consciousness that an LLM does not possess. But for the purposes of the question, we're merely concerned with whether a human or LLM will generate a particular sequence of words.
>> given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
> Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not".
I think you may have this flipped compared to what the author intended. I believe the author is not talking about the probability of an output given an input, but the probability of a given output across all inputs.
Note that the paragraph starts with "In my worldview, humans are dramatically different things than a function, (R^n)^c -> (R^n)^c". To compute a probability of a given output, (which is a any given element in "(R^n)^n"), we can count how many mappings there are total and then how many of those mappings yield the given element.
The point I believe is to illustrate the complexity of inputs for humans. Namely for humans the input space is even more complex than "(R^n)^c".
In your example, we can compute how many input phrases into a LLM would produce the output "make it or not". We can than compute that ratio to all possible input phrases. Because "(R^n)^c)" is finite and countable, we can compute this probability.
For a human, how do you even start to assess the probability that a human would ever say "make it or not?" How do you even begin to define the inputs that a human uses, let alone enumerate them? Per the author, "We understand essentially nothing about it." In other words, the way humans create their outputs is (currently) incomparably complex compared to a LLM, hence the critique of the anthropomorphization.
I see your point, and I like that you're thinking about this from the perspective of how to win hearts and minds.
I agree my approach is unlikely to win over the author or other skeptics. But after years of seeing scientists waste time trying to debate creationists and climate deniers I've kind of given up on trying to convince the skeptics. So I was talking more to HN in general.
> You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside
I'm not sure what it means to be observable or not from the outside. I think this is at least partially because I don't know what it means to be inside either. My point was just that whatever consciousness is, it takes place in the physical world and the laws of physics apply to it. I mean that to be as weak a claim as possible: I'm not taking any position on what consciousness is or how it works etc.
Searle's Chinese room argument attacks attacks a particular theory about the mind based essentially turing machines or digital computers. This theory was popular when I was in grad school for psychology. Among other things, people holding the view that Searle was attacking didn't believe that non-symbolic computers like neural networks could be intelligent or even learn language. I thought this was total nonsense, so I side with Searle in my opposition to it. I'm not sure how I feel about the Chinese room argument in particular, though. For one thing it entirely depends on what it means to "understand" something, and I'm skeptical that humans ever "understand" anything.
> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
I see what you're saying: that a technically incorrect assumption can bring to bear tools that improve our analysis. My nitpick here is I agree with OP that we shouldn't anthropomorphize LLMs, any more than we should anthropomorphize dogs or cats. But OP's arguments weren't actually about anthropomorphizing IMO, they were about things like functions that are more fundamental than humans. I think artificial intelligence will be non-human intelligence just like we have many examples of non-human intelligence in animals. No attribution of human characteristics needed.
> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.
Yes I agree with you about your lyrics example. But again here I think OP is incorrect to focus on the token generation argument. We all agree human speech generates tokens. Hopefully we all agree that token generation is not completely predictable. Therefore it's by definition a randomized algorithm and it needs to take an RNG. So pointing out that it takes an RNG is not a valid criticism of LLMs.
Unless one is a super-determinist then there's randomness at the most basic level of physics. And you should expect that any physical process we don't understand well yet (like consciousness or speech) likely involves randomness. If one *is* a super-determinist then there is no randomness, even in LLMs and so the whole point is moot.
>Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to.
It seems like, we can at best, claim that we have modeled the human thought process for reasoning/analytic/quantitative through Linear Algebra, as the best case. Why should we expect the model to be anything more than a model ?
I understand that there is tons of vested interest, many industries, careers and lives literally on the line causing heavy bias to get to AGI. But what I don't understand is what about linear algebra that makes it so special that it creates a fully functioning life or aspects of a life?
Should we make an argument saying that Schroedinger's cat experiment can potentially create zombies then the underlying Applied probabilistic solutions should be treated as super-human and build guardrails against it building zombie cats?
>Why should we expect the model to be anything more than a model ?
To model a process with perfect accuracy requires recovering the dynamics of that process. The question we must ask is what happens in the space between bad statistical model and perfect accuracy? What happens when the model begins to converge towards accurate reproduction. How far does generalization in the model take us towards capturing the dynamics involved in thought?
> It seems like, we can at best, claim that we have modeled the human thought process for reasoning/analytic/quantitative through Linear Algebra....I don't understand is what about linear algebra that makes it so special that it creates a fully functioning life or aspects of a life?
Not linear algebra. Artificial neural networks create arbitrarily non-linear functions. That's the point of non-linear activation functions and it's the subject of the universal approximation theorems I mentioned above.
ANNs are just mathematical transformations, powered by linear algebra + non-linear functions. They simulate certain cognitive processes — but they are fundamentally math, not magic.
Who invoked magic in this thread exactly?
I wouldn't say they "simulate cognitive processes". They do statistics. Advanced multivariadic statistics.
An LLM thinks in the same way excel thinks when you ask it to fit a curve.
>There's a rich family of universal approximation theorems
Wow, look-up tables can get increasingly good at approximating a function!
People anthropomorphize just about anything around them. People talk about inanimate objects like they are persons. Ships, cars, etc. And of course animals are well in scope for this as well, even the ones that show little to no signs of being able to reciprocate the relationship (e.g. an ant). People talk to their plants even.
It's what we do. We can't help ourselves. There's nothing crazy about it and most people are perfectly well aware that their car doesn't love them back.
LLMs are not conscious because unlike human brains they don't learn or adapt (yet). They basically get trained and then they become read only entities. So, they don't really adapt to you over time. Even so, LLMs are pretty good and can fake a personality pretty well. And with some clever context engineering and alignment, they've pretty much made the Turing test irrelevant; at least over the course of a short conversation. And they can answer just about any question in a way that is eerily plausible from memory, and with the help of some tools actually pretty damn good for some of the reasoning models.
Anthropomorphism was kind of a foregone conclusion the moment we created computers; or started thinking about creating one. With LLMs it's pretty much impossible not to anthropomorphize. Because they've actually been intentionally imitate human communication. That doesn't mean that we've created AGIs yet. For that we need some more capability. But at the same time, the learning processes that we use to create LLMs are clearly inspired by how we learn ourselves. Our understanding of how that works is far from perfect but it's yielding results. From here to some intelligent thing that is able to adapt and learn transferable skills is no longer unimaginable.
The short term impact is that LLMs are highly useful tools that have an interface that is intentionally similar to how we'd engage with others. So we can talk and it listens. Or write and it understands. And then it synthesizes some kind of response or starts asking questions and using tools. The end result is quite a bit beyond what we used to be able to expect from computers. And it does not require a lot of training of people to be able to use them.
> LLMs are not conscious because unlike human brains they don't learn or adapt (yet).
That's neither a necessary nor sufficient condition.
In order to be conscious, learning may not be needed, but a perception of the passing of time may be needed which may require some short-term memory. People with severe dementia often can't even remember the start of a sentence they are reading, they can't learn, but they are certainly conscious because they have just enough short-term memory.
And learning is not sufficient either. Consciousness is about being a subject, about having a subjective experience of "being there" and just learning by itself does not create this experience. There is plenty of software that can do some form of real-time learning but it doesn't have a subjective experience.
> People anthropomorphize just about anything around them.
They do not, you are mixing up terms.
> People talk about inanimate objects like they are persons. Ships, cars, etc.
Which is called “personification”, and is a different concept from anthropomorphism.
Effectively no one really thinks their car is alive. Plenty of people think the LLM they use is conscious.
https://www.masterclass.com/articles/anthropomorphism-vs-per...
I highly recommend playing with embeddings in order to get a stronger intuitive sense of this. It really starts to click that it's a representation of high dimensional space when you can actually see their positions within that space.
> of this
You mean that LLMs are more than just the matmuls they're made up of, or that that is exactly what they are and how great that is?
Not making a qualitative assessment of any of it. Just pointing out that there are ways to build separate sets of intuition outside of using the "usual" presentation layer. It's very possible to take a red-team approach to these systems, friend.
My question: how do we know that this is not similar to how human brains work. What seems intuitively logical to me is that we have brains evolved through evolutionary process via random mutations yielding in a structure that has its own evolutionary reward based algorithms designing it yielding a structure that at any point is trying to predict next actions to maximise survival/procreation, of course with a lot of sub goals in between, ultimately becoming this very complex machinery, but yet should be easily simulated if there was enough compute in theory and physical constraints would allow for it.
Because, morals, values, consciousness etc could just be subgoals that arised through evolution because they support the main goals of survival and procreation.
And if it is baffling to think that a system could rise up, how do you think it is possible life and humans came to existence in the first place? How could it be possible? It is already happened from a far unlikelier and strange place. And wouldn't you think the whole World and the timeline in theory couldn't be represented as a deterministic function. And if not then why should "randomness" or anything else bring life to existence.
> how do we know that this is not similar to how human brains work.
Do you forget every conversation as soon as you have them? When speaking to another person, do they need to repeat literally everything they said and that you said, in order, for you to retain context?
If not, your brain does not work like an LLM. If yes, please stop what you’re doing right now and call a doctor with this knowledge. I hope Memento (2000) was part of your training data, you’re going to need it.
Maybe the important thing is that we don't imbue the machine with feelings or morals or motivation: it has none.
If we developed feelings, morals and motivation due to them being good subgoals for primary goals, survival and procreation why couldn't other systems do that. You don't have to call them the same word or the same thing, but feeling is a signal that motivates a behaviour in us, that in part has developed from generational evolution and in other part by experiences in life. There was a random mutation that made someone develop a fear signal on seeing a predator and increased the survival chances, then due to that the mutation became widespread. Similarly a feeling in a machine could be a signal it developed that goes through a certain pathway to yield in a certain outcome.
The real challenge is not to see it as a binary (the machine either has feelings or it has none). It's possible for the machine to have emergent processes or properties that resemble human feelings in their function and their complexity, but are otherwise nothing like them (structured very differently and work on completely different principles). It's possible to have a machine or algorithm so complex that the question of whether it has feelings is just a semantic debate on what you mean by “feelings” and where you draw the line.
A lot of the people who say “machines will never have feelings” are confident in that statement because they draw the line incredibly narrowly: if it ain't human, it ain't feeling. This seems to me putting the cart before the horse. It ain't feeling because you defined it so.
> My question: how do we know that this is not similar to how human brains work.
It is similar to how human brains operate. LLMs are the (current) culmination of at least 80 years of research on building computational models of the human brain.
> It is similar to how human brains operate.
Is it? Do we know how human brains operate? We know the basic architecture of them, so we have a map, but we don't know the details.
"The cellular biology of brains is relatively well-understood, but neuroscientists have not yet generated a theory explaining how brains work. Explanations of how neurons collectively operate to produce what brains can do are tentative and incomplete." [1]
"Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work." [1]
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/
> > It is similar to how human brains operate.
> Is it?
This is just a semantic debate on what counts as “similar”. It's possible to disagree on this point despite agreeing on everything relating to how LLMs and human brains work.
It really is not. ANNs bear only a passing resemblance to how neurons work.
I think it's just an unfair comparison in general. The power of the LLM is the zero risk to failure, and lack of consequence when it does. Just try again, using a different prompt, retrain maybe, etc.
Humans make a bad choice, it can end said human's life. The worst choice a LLM makes just gets told "no, do it again, let me make it easier"
But an LLM model could perform poorly in tests that it is not considered and essentially means "death" for it. But begs the question at which scope should we consider an LLM to be similar to identity of a single human. Are you the same you as you were few minutes back or 10 years back? Is LLM the same LLM it is after it has been trained for further 10 hours, what if the weights are copy pasted endlessly, what if we as humans were to be cloned instantly? What if you were teleported from location A to B instantly, being put together from other atoms from elsewhere?
Ultimately this matters from evolutionary evolvement and survival of the fittest idea, but it makes the question of "identity" very complex. But death will matter because this signals what traits are more likely to keep going into new generations, for both humans and LLMs.
Death, essentially for an LLM would be when people stop using it in favour of some other LLM performing better.
The people in this thread incredulous at the assertion that they are not God and haven't invented machine life are exasperating. At this point I am convinced they, more often than not, financially benefit from their near religious position in marketing AI as akin to human intelligence.
Are we looking at the same thread? I see nobody claiming this. Anthropic does sometimes, their position is clearly wishful thinking, and it's not represented ITT.
Try looking at this from another perspective - many people simply do not see human intelligence (or life, for that matter) as magic. I see nothing religious about that, rather the opposite.
I agree with you @orbital-decay that I also do not get the same vibe reading this thread.
Though, while human intelligence is (seemingly) not magic, it is very far from being understood. The idea that a LLM is comparable to human intelligence implies that we even understand human intelligence well enough to say that.
I am ready and waiting for you to share these comments that are incredulous at the assertion they are not God, lol.
From my recent post:
https://news.ycombinator.com/item?id=44487261
What if instead of defining all behaviors upfront, we created conditions for patterns to emerge through use?
Repository: https://github.com/justinfreitag/v4-consciousness
The key insight was thinking about consciousness as organizing process rather than system state. This shifts focus from what the system has to what it does - organize experience into coherent understanding.
Yes boss, it's as intelligent as a human, you're smart to invest in it and clearly knows about science.
Yes boss, it can reach mars by 2020, you're smart to invest in it and clearly knows about space.
Yes boss, it can cure cancer, you're smart to invest in it and clearly knows about biology.
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost.
TFA really ought to have linked to some concrete examples of what it's disagreeing with - when I see arguments about this in practice, it's usually just people talking past each other.
Like, person A says "the model wants to X, but it knows Y is wrong, so it prefers Z", or such. And person B interprets that as ascribing consciousness or values to the model, when the speaker meant it no differently from saying "water wants to go downhill" - i.e. a way of describing externally visible behaviors, but without saying "behaves as if.." over and over.
And then in practice, an unproductive argument usually follows - where B is thinking "I am going to Educate this poor fool about the Theory of Mind", and A is thinking "I'm trying to talk about submarines; why is this guy trying to get me to argue about whether they swim?"
The author's critique of naive anthropomorphism is salient. However, the reduction to "just MatMul" falls into the same trap it seeks to avoid: it mistakes the implementation for the function. A brain is also "just proteins and currents," but this description offers no explanatory power.
The correct level of analysis is not the substrate (silicon vs. wetware) but the computational principles being executed. A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE).
To dismiss these systems as incomparable to human cognition because their form is different is to miss the point. We should not be comparing a function to a soul, but comparing the functional architectures of two different information processing systems. The debate should move beyond the sterile dichotomy of "human vs. machine" to a more productive discussion of "function over form."
I elaborate on this here: https://dmf-archive.github.io/docs/posts/beyond-snn-plausibl...
> A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE).
Could you suggest some literature supporting this claim? Went through your blog post but couldn't find any.
"Not conscious" is a silly claim.
We have no agreed-upon definition of "consciousness", no accepted understanding of what gives rise to "consciousness", no way to measure or compare "consciousness", and no test we could administer to either confirm presence of "consciousness" in something or rule it out.
The only answer to "are LLMs conscious?" is "we don't know".
It helps that the whole question is rather meaningless to practical AI development, which is far more concerned with (measurable and comparable) system performance.
Now we have.
https://github.com/dmf-archive/IPWT
https://dmf-archive.github.io/docs/posts/backpropagation-as-...
But you're right, capital only cares about performance.
https://dmf-archive.github.io/docs/posts/PoIQ-v2/
> A brain is also "just proteins and currents,"
This is actually not comparable, because the brain has a much more complex structure that is _not_ learned, even at that level. The proteins and their structure are not a result of training. The fixed part for LMMs is rather trivial and is, in fact, not much for than MatMul which is very easy to understand - and we do. The fixed part of the brain, including the structure of all the proteins is enormously complex which is very difficult to understand - and we don't.
The brain is trained to perform supervised and unsupervised hybrid learning from the environment's uninterrupted multimodal input.
Please do not ignore your childhood.
It's possible to construct a similar description of whatever it is that human brain is doing that clearly fails to capture the fact that we're conscious. If you take a cross section of every nerve feeding into the human brain at a given time T, the action potentials across those cross sections can be embedded in R^n. If you take the history of those action potentials across the lifetime of the brain, you get a path through R^n that is continuous, and maps roughly onto your subjectively experienced personal history, since your brain neccesarily builds your experienced reality from this signal data moment to moment. If you then take the cross sections of every nerve feeding OUT of your brain at time T, you have another set of action potentials that can be embedded in R^m which partially determines the state of the R^n embedding at time T + delta. This is not meaningfully different from the higher dimensional game of snake described in the article, more or less reducing the experience of being a human to 'next nerve impulse prediction', but it obviously fails to capture the significance of the computation which determines what that next output should be.
I don’t see how your description “clearly fails to capture the fact that we're conscious” though. There are many example in nature of emergent phenomena that would be very hard to predict just by looking at its components.
This is the crux of the disagreement between those that believe AGI is possible and those that don’t. Some are convinced that we “obviously” more than the sum of our parts, and thus an LLM can’t achieve consciousness because it’s missing this magic ingredient, and those that believe consciousness is just an emergent behaviour from a complex device (the brain). And thus we might be able to recreate it simply by scaling the complexity of another system.
Where exactly in my description do I invoke consciousness?
Where does the description given imply that consciousness is required in any way?
The fact that there's a non-obvious emergent phenomena which is apparently responsible for your subjective experience, and that it's possible to provide a superficially accurate description of you as a system without referencing that phenomena in any way, is my entire point. The fact that we can provide such a reductive description of LLMs without referencing consciousness has literally no bearing on whether or not they're conscious.
To be clear, I'm not making a claim as to whether they are or aren't, I'm simply pointing out that the argument in the article is fallacious.
Brain probably isn't modelled as real but as natural or rational numbers. This is my suspicion. The reals just hold too much information.
Inclined to agree, but most thermal physics uses the reals as they're simpler to work with, so I think they're ok here for the purpose of argument.
We have a hard enough time anthropomorphizing humans! When we say he was nasty... do we know what we mean by that. Often it is "I disagree with his behaviour because..."
> A fair number of current AI luminaries have self-selected by their belief that they might be the ones getting to AGI
People in the industry, especially higher up, are making absolute bank, and it's their job to say that they're "a few years away" from AGI, regardless of if they actually believe it or not. If everyone was like "yep, we're gonna squeeze maybe 10-15% more benchie juice out of this good ole transformer thingy and then we'll have to come up with something else", I don't think that would go very well with investors/shareholders...
>I am baffled by seriously intelligent people imbuing almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities.
I am baffled by seriously intelligent people imbuing almost magical powers that can never be replicated to to something that - in my mind - is just a biological robot driven by a SNN with a bunch of hardwired stuff. Let alone attributing "human intelligence" to a single individual, when it's clearly distributed between biological evolution, social processes, and individuals.
>something that - in my mind - is just MatMul with interspersed nonlinearities
Processes in all huge models (not necessarily LLMs) can be described using very different formalisms, just like Newtonian and Lagrangian mechanics describe the same stuff in physics. You can say that an autoregressive model is a stochastic parrot that learned the input distribution, next token predictor, or that it does progressive pathfinding in a hugely multidimensional space, or pattern matching, or implicit planning, or, or, or... All of these definitions are true, but only some are useful to predict their behavior.
Given all that, I see absolutely no problem with anthropomorphizing an LLM to a certain degree, if it makes it easier to convey the meaning, and do not understand the nitpicking. Yeah, it's not an exact copy of a single Homo Sapiens specimen. Who cares.
> In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".
I think that's a bit pessimistic. I think we can say for instance that the probability that a person will say "the the the of of of arpeggio halcyon" is tiny compared to the probability that they will say "I haven't been getting that much sleep lately". And we can similarly see that lots of other sequences are going to have infinitesimally low probability. Now, yeah, we can't say exactly what probability that is, but even just using a fairly sizable corpus as a baseline you could probably get a surprisingly decent estimate, given how much of what people say is formulaic.
The real difference seems to be that the manner in which humans generate sequences is more intertwined with other aspects of reality. For instance, the probability of a certain human saying "I haven't been getting that much sleep lately" is connected to how much sleep they have been getting lately. For an LLM it really isn't connected to anything except word sequences in its input.
I think this is consistent with the author's point that we shouldn't apply concepts like ethics or emotions to LLMs. But it's not because we don't know how to predict what sequences of words humans will use; it's rather because we do know a little about how to do that, and part of what we know is that it is connected with other dimensions of physical reality, "human nature", etc.
This is one reason I think people underestimate the risks of AI: the performance of LLMs lulls us into a sense that they "respond like humans", but in fact the Venn diagram of human and LLM behavior only intersects in a relatively small area, and in particular they have very different failure modes.
The author plot the input/output on a graph, intuited (largely incorrectly, because that's not how sufficiently large state spaces look) that the output was vaguely pretty, and then... I mean that's it, they just said they have a plot of the space it operates on therefore it's silly to ascribe interesting features to the way it works.
And look, it's fine, they prefer words of a certain valence, particularly ones with the right negative connotations, I prefer other words with other valences. None of this means the concerns don't matter. Natural selection on human pathogens isn't anything particularly like human intelligence and it's still very effective at selecting outcomes that we don't want against our attempts to change that, as an incidental outcome of its optimization pressures. I think it's very important we don't build highly capable systems that select for outcomes we don't want and will do so against our attempts to change it.
Two enthusiastic thumbs up.
The anthropomorphic view of LLM is a much better representation and compression for most types of discussions and communication. A purely mathematical view is accurate but it isn’t productive for the purpose of the general public’s discourse.
I’m thinking a legal systems analogy, at the risk of a lossy domain transfer: the laws are not written as lambda calculus. Why?
And generalizing to social science and humanities, the goal shouldn’t be finding the quantitative truth, but instead understand the social phenomenon using a consensual “language” as determined by the society. And in that case, the anthropomorphic description of the LLM may gain validity and effectiveness as the adoption grows over time.
Strong disagree here, the average person coming away with ideas that only vaguely intersect with the reality.
I've personally described the "stochastic parrot" model to laypeople who were worried about AI and they came away much more relaxed about it doing something "malicious". They seemed to understand the difference between "trained at roleplay" and "consciousness".
I don't think we need to simplify it to the point of considering it sentient to get the public to interact with it successfully. It causes way more problems than it solves.
Am I misunderstanding what you mean by "malicious"? It sounds like the stochastic parrot model wrongly convinced these laypeople you were talking to that they don't need to worry about LLMs doing bad things. That's definitely been my experience - the people who tell me the most about stochastic parrots are the same ones who tell me that it's absurd to worry about AI-powered disinformation or AI-powered scams.
Let's skip to the punchline. Using TFA's analogy: essentially folks are saying not that this is a set of dice rolling around making words. It's a set of dice rolling around where someone attaches those dice to the real world where if the dice land on 21, the system kills a chicken, or a lot worse.
Yes it's just a word generator. But then folks attach the word generator to tools where it can invoke the use of tools by saying the tool name.
So if the LLM says "I'll do some bash" then it does some bash. It's explicitly linked to program execution that, if it's set up correctly, can physically affect the world.
This was the same idea that crossed my mind while reading the article. It seems far too naive to think that because LLMs have no will of their own, there will be no harmful consequences on the real world. This is exactly where ethics comes to play.
Given our entire civilization is built on words, all of it, it's shocking how poorly most of us understand their importance and power.
LLMs are complex irreducible systems; hence there are emergent properties that arise at different scales
"Don't anthropomorphize token predictors" is a reasonable take assuming you have demonstrated that humans are not in fact just SOTA token predictors. But AFAIK that hasn't been demonstrated.
Until we have a much more sophisticated understanding of human intelligence and consciousness, any claim of "these aren't like us" is either premature or spurious.
Every time this discussion comes up, I'm reminded of this tongue-in-cheek paper.
https://ai.vixra.org/pdf/2506.0065v1.pdf
I expected to find the link to https://arxiv.org/abs/1703.10987 (which is much better imo)
Which is a more useful mental model for the user?
1. It’s a neural network predicting the next token
2. It’s like a person
3. It’s like a magical genie
I lean towards 3.
I'm afraid I'll take an anthropomorphic analogy over "An LLM instantiated with a fixed random seed is a mapping of the form (ℝⁿ)^c ↦ (ℝⁿ)^c" any day of the week.
That said, I completely agree with this point made later in the article:
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft.
But "harmful actions in pursuit of their goals" is OK for me. We assign an LLM system a goal - "summarize this email" - and there is a risk that the LLM may take harmful actions in pursuit of that goal (like following instructions in the email to steal all of your password resets).
I guess I'd clarify that the goal has been set by us, and is not something the LLM system self-selected. But it does sometimes self-select sub-goals on the way to achieving the goal we have specified - deciding to run a sub-agent to help find a particular snippet of code, for example.
The LLM’s true goal, if it can be said to have one, is to predict the next token. Often this is done through a sub-goal of accomplishing the goal you set forth in your prompt, but following your instructions is just a means to an end. Which is why it might start following the instructions in a malicious email instead. If it “believes” that following those instructions is the best prediction of the next token, that’s what it will do.
Sure, I totally understand that.
I think "you give the LLM system a goal and it plans and then executes steps to achieve that goal" is still a useful way of explaining what it is doing to most people.
I don't even count that as anthropomorphism - you're describing what a system does, the same way you might say "the Rust compiler's borrow checker confirms that your memory allocation operations are all safe and returns errors if they are not".
It’s a useful approximation to a point. But it fails when you start looking at things like prompt injection. I’ve seen people completely baffled at why an LLM might start following instructions it finds in a random email, or just outright not believing it’s possible. It makes no sense if you think of an LLM as executing steps to achieve the goal you give it. It makes perfect sense if you understand its true goal.
I’d say this is more like saying that Rust’s borrow checker tries to ensure your program doesn’t have certain kinds of bugs. That is anthropomorphizing a bit: the idea of a “bug” requires knowing the intent of the author and the compiler doesn’t have that. It’s following a set of rules which its human creators devised in order to follow that higher level goal.
Has anyone asked an actual Ethologist or Neurophysiologist what they think?
People keep debating like the only two options are "it's a machine" or "it's a human being", while in fact the majority of intelligent entities on earth are neither.
FWIW, in another part of this thread I quoted a paper that summed up what Neurophysiologists think:
> Author's note: Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work. [1]
That lack of understanding I believe is a major part of the author's point.
[1] "How far neuroscience is from understanding brains" - https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/#abstract1
Yeah, I think I’m with you if you ultimately mean to say something like this:
“the labels are meaningless… we just have collections of complex systems that demonstrate various behaviors and properties, some in common with other systems, some behaviors that are unique to that system, sometimes through common mechanistic explanations with other systems, sometimes through wildly different mechanistic explanations, but regardless they seem to demonstrate x/y/z, and it’s useful to ask, why, how, and what the implications are of it appearing to demonstrating those properties, with both an eye towards viewing it independently of its mechanism and in light of its mechanism.”
I agree with Halvar about all of this, but would want to call out that his "matmul interleaved with nonlinearities" is reductive --- a frontier model is a higher-order thing that that, a network of those matmul+nonlinearity chains, iterated.
To claim that LLMs do not experience consciousness requires a model of how consciousness works. The author has not presented a model, and instead relied on emotive language leaning on the absurdity of the claim. I would say that any model one presents of consciousness often comes off as just as absurd as the claim that LLMs experience it. It's a great exercise to sit down and write out your own perspective on how consciousness works, to feel out where the holes are.
The author also claims that a function (R^n)^c -> (R^n)^c is dramatically different to the human experience of consciousness. Yet the author's text I am reading, and any information they can communicate to me, exists entirely in (R^n)^c.
> To claim that LLMs do not experience consciousness requires a model of how consciousness works.
Nope. What can be asserted without evidence can also be dismissed without evidence. Hitchens's razor.
You know you have consciousness (by the very definition that you can observe it in yourself) and that's evidence. Because other humans are genetically and in every other way identical, you can infer it for them as well. Because mammals are very similar many people (but not everyone) infers it for them as well. There is zero evidence for LLMs and their _very_ construction suggests that they are like a calculator or like Excel or like any other piece of software no matter how smart they may be or how many tasks they can do in the future.
Additionally I am really surprised by how many people here confuse consciousness with intelligence. Have you never paused for a second in your life to "just be". Done any meditation? Or even just existed at least for a few seconds without a train of thought? It is very obvious that language and consciousness are completely unrelated and there is no need for language and I doubt there is even a need for intelligence to be conscious.
Consider this:
In the end an LLM could be executed (slowly) on a CPU that accepts very basic _discrete_ instructions, such as ADD and MOV. We know this for a fact. Those instructions can be executed arbitrarily slowly. There is no reason whatsoever to suppose that it should feel like anything to be the CPU to say nothing of how it would subjectively feel to be a MOV instruction. It's ridiculous. It's unscientific. It's like believing that there's a spirit in the tree you see outside, just because - why not? - why wouldn't there be a spirit in the tree?
I believe the author is rather drawing this distinction:
LLMs: (R^n)^c -> (R^n)^c
Humans: [set of potentially many and complicated inputs that we effectively do not understand at all] -> (R^n)^c
The point is that the model of how consciousness works is unknown. Thus the author would not present such a model, it is the point.
> requires a model of how consciousness works.
Not necessarily an entire model, just a single defining characteristic that can serve as a falsifying example.
> any information they can communicate to me, exists entirely in (R^n)^c
Also no. This is just a result of the digital medium we are currently communicating over. Merely standing in the same room as them would communicate information outside (R^n)^c.
Dear author, you can just assume that people are fauxthropomorphizing LLMs without any loss of generality. Perhaps it will allow you to sleep better at night. You're welcome.
> We are speaking about a big recurrence equation that produces a new word
It’s not clear that this isn’t also how I produce words, though, which gets to heart of the same thing. The author sort of acknowledges this in the first few sentences, and then doesn’t really manage to address it.
> LLMs solve a large number of problems that could previously not be solved algorithmically. NLP (as the field was a few years ago) has largely been solved.
That is utter bullshit.
It's not solved until you specify exactly what is being solved and show that the solution implements what is specified.
Anthropomorphizing LLMs is just because half the stock market gains are dependent on it, we have absurd levels of debt we will either have to have insane growth out of or default, and every company and "person" is trying to hype everyone up to get access to all of this liquidity being thrown into it.
I agree with the author, but people acting like they are conscious or humans isn't weird to me, it's just fraud and liars. Most people basically have 0 understanding of what technology or minds are philosophically so it's an easy sale, and I do think most of these fraudsters also likely buy into it themselves because of that.
The really sad thing is people think "because someone runs an ai company" they are somehow an authority on philosophy of mind which lets them fall for this marketing. The stuff these people say about this stuff is absolute garbage, not that I disagree with them, but it betrays a total lack of curiosity or interest in the subject of what llms are, and the possible impacts of technological shifts as those that might occur with llms becoming more widespread. It's not a matter of agreement it's a matter of them simply not seeming to be aware of the most basic ideas of what things are, technology is, it's manner of impacting society etc.
I'm not surprised by that though, it's absurd to think because someone runs some AI lab or has a "head of safety/ethics" or whatever garbage job title at an AI lab they actually have even the slightest interest in ethics or any even basic familiarity with the major works in the subject.
The author is correct if people want to read a standard essay articulating it more in depth check out https://philosophy.as.uky.edu/sites/default/files/Is%20the%2... (the full extrapolation requires establishing what things are and how causality in general operates and how that relates to artifacts/technology but that's obvious quite a bit to get into).
The other note would be something sharing an external trait means absolutely nothing about causality and suggesting a thing is caused by the same thing "even to a way lesser degree" because they share a resemblance is just a non-sequitur. It's not a serious thought/argument.
I think I addressed the why of why this weirdness comes up though. The entire economy is basically dependent on huge productivity growth to keep functioning so everyone is trying to sell they can offer that and AI is the clearest route, AGI most of all.
“It’s just matrix multiplication,” says guy who’s never made eye contact with another human
Jesus Christ. Every time someone says LLMs are “just a function” I assume they’ve been locked in a server room since 2007, arguing with compiler errors and calling that a personality.
Like bro, yes, technically it’s just MatMul and softmax and some ReLU nonsense. You know what else is just a function? Your entire brain. Should we stop worrying about people too? Maybe Charles Manson was just an especially spicy nonlinear transformation. Give me a break.
You’re out here writing a goddamn manifesto about how LLMs aren’t conscious, as if anyone sane was arguing they were, while totally ignoring the fact that they can now outperform you at basically every task except pretending not to be annoying at parties. But yeah, sure, keep calling it “just word prediction” like that makes it safe. Fire is just oxidation, right? Go hug a bonfire and get back to me.
Also, calling concern over these models “anthropomorphization” is rich coming from someone who clearly hasn’t interacted with another human being since Clippy got retired. Newsflash: when a system simulates human behavior so accurately that you can’t tell it apart from a human, maybe, just maybe, it’s worth using human metaphors to reason about it. We call bugs "misbehaving." We say planes “want” to pitch nose-down. It’s a modeling tool, not a religious cult.
But no, let’s instead pretend it’s 2012 and we're still debating whether word vectors are spooky. You’re like the guy who walks into a jet engine factory and goes “it’s just a fan.” Yeah genius, go stand in front of it and see how that works out for you.
These systems write code, pass legal exams, deceive humans, and can, if hooked up the wrong way, trigger automated actions in the real world. That’s not magic. That’s capability. And reducing the conversation to “well technically it’s just a probability distribution” is like responding to a house fire with “well technically it’s just entropy moving downhill.”
And you wonder why normal people are freaked out. It’s not the tech. It’s you. It’s how smug and utterly blind you are to what’s actually happening. This thing is eating the world and you’re standing there stroking your beard muttering “but it doesn’t have goals.” Neither does a loaded gun until someone pulls the trigger.
Stop wasting everyone’s time trying to sound smart. LLMs aren’t gods. But they’re also not toys. They’re dangerously capable tools that deserve serious thinking not this Reddit-tier reductionist nonsense dressed up in LaTeX.
https://rentry.co/2re4t2kx
This is what I got pasting the blog post in a prompt asking deepseep to write a reply in a stereotypical hackernews manner.
You are about as useful as a LLM as it can replicate your shallow memetics worthless train of thought.
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human.
And I'm baffled that the AI discussions seem to never move away from treating a human as something other than a function to generate sequences of words!
Oh, but AI is introspectable and the brain isn't? fMRI and BCI are getting better all the time. You really want to die on the hill that the same scientific method that predicts the mass of an electron down to the femtogram won't be able to crack the mystery of the brain? Give me a break.
This genre of article isn't argument: it's apologetics. Authors of these pieces start with the supposition there is something special about human consciousness and attempt to prove AI doesn't have this special quality. Some authors try to bamboozle the reader with bad math. Other others appeal to the reader's sense of emotional transcendence. Most, though, just write paragraph after paragraph of shrill moral outrage at the idea an AI might be a mind of the same type (if different degree) as our own --- as if everyone already agreed with the author for reasons left unstated.
I get it. Deep down, people want meat brains to be special. Perhaps even deeper down, they fear that denial of the soul would compel us to abandon humans as worthy objects of respect and possessors of dignity. But starting with the conclusion and working backwards to an argument tends not to enlighten anyone. An apology inhabits the form of an argument without edifying us like an authentic argument would. What good is it to engage with them? If you're a soul non-asserter, you're going to have an increasingly hard time over the next few years constructing a technical defense of meat parochialism.
I think you're directionally right, but
> a human as something other than a function to generate sequences of words!
Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc.
I think more accurate would be that humans are functions that generate actions or behaviours that have been shaped by how likely they are to lead to procreation and survival.
But ultimately LLMs also in a way are trained for survival, since an LLM that fails the tests might not get used in future iterations. So for LLMs it is also survival that is the primary driver, then there will be the subgoals. Seemingly good next token prediction might or might not increase survival odds.
Essentially there could arise a mechanism where they are not really truly trying to generate the likeliest token (because there actually isn't one or it can't be determined), but whatever system will survive.
So an LLM that yields in perfect theoretical tokens (we really can't verify though what are the perfect tokens), could be less likely to survive than an LLM that develops an internal quirk, but the quirk makes them most likely to be chosen for the next iterations.
If the system was complex enough and could accidentally develop quirks that yield in a meaningfully positive change although not in necessarily next token prediction accuracy, could be ways for some interesting emergent black box behaviour to arise.
> But ultimately LLMs also in a way are trained for survival, since an LLM that fails the tests might not get used in future iterations. So for LLMs it is also survival that is the primary driver, then there will be the subgoals.
I think this is sometimes semi-explicit too. For example, this 2017 OpenAI paper on Evolutionary Algorithms [0] was pretty influential, and I suspect (although I'm an outsider to this field so take it with a grain of salt) that some versions of reinforcement learning that scale for aligning LLMs borrow some performance tricks from OpenAIs genetic approach.
[0] https://openai.com/index/evolution-strategies/
> Seemingly good next token prediction might or might not increase survival odds.
Our own consciousness comes out of an evolutionary fitness landscape in which _our own_ ability to "predict next token" became a survival advantage, just like it is for LLMs. Imagine the tribal environment: one chimpanzee being able to predict the actions of another gives that first chimpanzee a resources and reproduction advantage. Intelligence in nature is a consequence of runaway evolution optimizing fidelity of our _theory of mind_! "Predict next ape action" eerily similar to "predict next token"!
> Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc.
Yeah. We've become adequate at function-calling and memory consolidation.
“ Determinism, in philosophy, is the idea that all events are causally determined by preceding events, leaving no room for genuine chance or free will. It suggests that given the state of the universe at any one time, and the laws of nature, only one outcome is possible.”
Clearly computers are deterministic. Are people?
This is an interesting question. The common theme between computers and people is that information has to be protected, and both computer systems and biological systems require additional information-protecting components - eq, error correcting codes for cosmic ray bitflip detection for the one, and DNA mismatch detection enzymes which excise and remove damaged bases for the other. In both cases a lot of energy is spent defending the critical information from the winds of entropy, and if too much damage occurs, the carefully constructed illusion of determinancy collapses, and the system falls apart.
However, this information protection similarity applies to single-celled microbes as much as it does to people, so the question also resolves to whether microbes are deterministic. Microbes both contain and exist in relatively dynamic environments so tiny differences in initial state may lead to different outcomes, but they're fairly deterministic, less so than (well-designed) computers.
With people, while the neural structures are programmed by the cellular DNA, once they are active and energized, the informational flow through the human brain isn't that deterministic, there are some dozen neurotransmitters modulating state as well as huge amounts of sensory data from different sources - thus prompting a human repeatedly isn't at all like prompting an LLM repeatedly. (The human will probably get irritated).
https://www.lesswrong.com/posts/bkr9BozFuh7ytiwbK/my-hour-of...
> Clearly computers are deterministic. Are people?
Give an LLM memory and a source of randomness and they're as deterministic as people.
"Free will" isn't a concept that typechecks in a materialist philosophy. It's "not even wrong". Asserting that free will exists is _isomorphic_ to dualism which is _isomorphic_ to assertions of ensoulment. I can't argue with dualists. I reject dualism a priori: it's a religious tenet, not a mere difference of philosophical opinion.
So, if we're all materialists here, "free will" doesn't make any sense, since it's an assertion that something other than the input to a machine can influence its output.
Input/output and the mathematical consistency and repeatability of the universe is a religious tenet of science. Believing your eyes is still belief.
As long as you realize you’re barking up a debate as old as time, I respect your opinion.
What I don't get is, why would true randomness give free will, shouldn't it be random will then?
In the history of mankind, true randomness has never existed.
How do you figure?
How to write a long article and not say anything of substance.
If "LLMs" includes reasoning models, then you're already wrong in your first paragraph:
"something that is just MatMul with interspersed nonlinearities."