It's unfortunate that there's so little (none in the article, just 1 comment here as of this writing) mention of the Turing Test. The whole premise of the paper that introduced that was that "do machines think" is such a hard question to define that you have to frame the question differently. And it's ironic that we seem to talk about the Turing Test less than ever now that systems almost everyone can access can arguably pass it now.
> “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.” ~ Edsger W. Dijkstra
The point of the Turing Test is that if there is no extrinsic difference between a human and a machine the intrinsic difference is moot for practical purposes. That is not an argument to whether a machine (with linear algebra, machine learning, large language models, or any other method) can think or what constitutes thinking or consciousness.
The Chinese Room is a pretty useless thought exercise I think. It's an example which if you believe machines can't think seems like an utterly obvious result, and if you believe machines can think it's just obviously wrong.
>And it's ironic that we seem to talk about the Turing Test less than ever now that systems almost everyone can access can arguably pass it now.
Has everyone hastily agreed that it has been passed? Do people argue that a human can't figure out it's talking to an LLM if the user is aware that LLMs exist in the world and is aware of their limitations and that the chat log is able to extend to infinity ( "infinity" is a proxy here for any sufficient time, it could be minutes, days, months, or years)?
In fact, it is blindly easy for these systems to fail the Turing test at the moment. No human would have the patience to continue a conversation indefinitely without telling the person on the other side to kindly fuck off.
I'm pretty sure a set of workshops isn't ACTUALLY going to solve a problem that philosophers have been at each other's throats for for the past half century.
But BOY does it get people talking!
Both sides of the debate have capital-O Opinions, and how else did you want to drum up interest for a set of mathematics workshops. O:-)
Yes, that word 'think' is a hell of a distraction. We've replaced that bit of the title with a less provocative / more interesting phrase from the article.
Hopefully we can talk about the actual math and stuff (although the article doesn't go into much of that).
artificial intelligence in place of artificial knowledge is confusing distraction too, at least artificial is a good reminder since beginning
(IMHO its not provocative but well catching a point.. about so called "intelligence" - what if we could look for intelligent knowledge - made in not statistic but semantic(?) and converging way - instead - of being distracted, afraid and.. outpriced, peanuts for the essence it doesn't have ?)
I wish I would've learned about ANNs in elementary school. It looks like a worthwhile and cool lesson package, if only they'd do away with the idiotic dogma...
I love the idea of educating students on the math behind AI to demystify them. But I think it's a little weird to assert "AI is not magic and AI systems do not think. It’s just maths." Equivalent statements could be made about how human brains are not magic, just biology - yet I think we still think.
I agree saying "they don't think" and leaving it at that isn't particularly useful or insightful, it's like saying "submarines don't swim" and refusing to elaborate further. It can be useful if you extend it to "they don't think like you do". Concepts like finite context windows, or the fact that the model is "frozen" and stateless, or the idea that you can transfer conversations between models are trivial if you know a bit about how LLMs work, but extremely baffling otherwise.
> or the fact that the model is "frozen" and stateless,
much like a human adult. Models get updated at a slower frequency than humans. AI systems have access to fetch new information and store it for context.
> or the idea that you can transfer conversations between models are trivial
because computers are better-organized than humanity.
it doesn't sound like you really understand what these statements mean. if LLMs are like any humans it's those with late stage dementia, not healthy adults
My context window is about a day. I can remember what I had for lunch today, and sometimes what I had for lunch yesterday. Beyond that, my lunches are gone from my context window and are only in my training data. I have vague ideas about what dishes I ate, but don't remember what days specifically. If I had to tell you what separate dishes I ate in the same meal, I don't have specific memories of that. I remember I ate fried plantains, and I ate beans & rice. I assume they were on the same day because they are from the same cuisine, and am confident enough that I would bet money on it, but I don't know for certain.
One of my earliest memories is of painting a ceramic mug when I was about 3 years old. The only reason I remember it is because every now and then I think about what my earliest memory is, and then I refresh my memory of it. I used to remember a few other things from when I was slightly older, but no longer do, because I haven't had reasons to think of them.
I don't think humans have specific black and white differences between types of knowledge that way LLMs do, but there is definitely a lot of behavior that is similar to context window vs training data (and a gradient in between). We remember recent things a lot better than less recent things. The quantity of stuff we can remember in our "working memory" is approximately finite. If you try to hold a complex thought in your mind, you can probably do that indefinitely, but if you then try to hold a second equally complex thought as well, you'll often lose the details of the first thought and need to reread or rederive those details.
A lot of people genuinely can't remember what they did an hour ago, but to be very clear you're implying that an LLM can't "remember" something from an hour, or three hours ago, when it's the opposite.
I can restart a conversation with an LLM 15 days later and the state is exactly as it was.
Can't do that with a human.
The idea that humans have a longer, more stable context window than LLM's, CAN or is even LIKELY to be true given certain activities but please let's be honest about this.
If you talk to someone for an hour about a technical conversation I would guesstimate that 90% of humans would immediately start to lose track of details in about 10 minutes. So they write things down, or they mentally repeat things to themselves they know or have recognized they keep forgetting.
I know this because it's happened continually in tech companies decade after decade.
LLM's have already passed the Turing test. They continue to pass it. They fool and outsmart people day after day.
I'm no fan of the hype AI is receiving, especially around overstating its impact in technical domains, but pretending that LLM's can't or don't consistently perform better than most human adults on a variety of different activities is complete non-sense.
It's just provencial nonsense, there's no sound reasoning to it. Reductionism being taken and used as a form of refutation is a pretty common cargo culting behavior I've found.
Overwhelmingly, I just don't think the majority of human beings have the mental toolset to work with ambiguous philosophical contexts. They'll still try though, and what you get out of that is a 4th order baudrillardian simulation of reason.
"Just" is used here as a reductive device. You reduce others to a few sentences.
Sentences constructed of words and representations of ideas defined long before you existed. I question whether you can work with ambiguous contexts as you have had the privilege of them being laid out in language for you already by the time you were born.
From my reference frame you appear to merely be circumlocuting from memory, and become the argument you make about others.
AI and brains can do some, AI and brains definitely provably cannot do others, some others are untestable at present, and nobody really knows enough about what human brains do to be able to tell if or when some existing or future AI can do whatever is needed for the stuff we find special about ourselves.
A lot of people use different definitions, and respond to anyone pointing this out by denying the issue and claiming their own definition is the only sensible one and "obviously" everyone else (who isn't a weird pedant) uses it.
The definition of "thinking" in any of the parent comments or TFA is actually not defined. Like literally no statements are made about what is being tested.
So, if we had that we could actually discuss it. Otherwise it's just opinions about what a person believes thinking is, combined with what LLMs are doing + what the person believes they themselves do + what they believe others do. It's entirely subjective with very low SNR b/c of those confounding factors.
There are people who insist that the halting problem "proves" that machines will never be able to think. That this means they don't understand the difference between writing down (or generating a proof of) the halting problem and the implications of the halting problem, does not stop them from using it.
I don't know that I agree that computation is a variety of thinking. It's certainly influenced by thinking, but I think of thinking as more the thing you do before, after, and in-between the computation, not the actual computation itself.
Statements like "it is bound by the laws of physics" are not "verifiable" by your definition, and yet we safely assume it is true of everything. Everything except the human brain, that is, for which wild speculation that it may be supernatural is seemingly considered rational discussion so long as it satisfies people's needs to believe that they are somehow special in the universe.
I think what many are saying is that of all the things we know best, it's going to be the machines we build and their underlying principles.
We don't fully understand how brains work, but we know brains don't function like a computer. Why would a computer be assumed to function like a brain in any way, even in part, without evidence and just hopes based on marketing? And I don't just mean consumer marketing, but marketing within academia as well. For example, names like "neural networks" have always been considered metaphorical at best.
What has it got to do with anything whether brains function like computers? This is only relevant if you define thinking as something only the brain can do, and then nothing that doesn't work like a brain can think. This would be like defining flight as "what birds do" and then saying airplanes can't fly because they don't work like birds.
And then what do you even mean by "a computer?" This falls into the same trap because it sounds like your statement that brains don't function like a computer is really saying "brains don't function like the computers I am familiar with." But this would be like saying quantum computers aren't computers because they don't work like classical computers.
To use your own example, it's relevant because the definition of "flight" that we apply to planes is not as versatile as the one we apply to birds.
To put this in terms of "results", because that's what your way of thinking insists upon, a plane does not take off and land the way a bird does. This limits a plane's practicality to such an extent that a plane is useless for transportation without all the infrastructure you're probably ignoring with your argument. You might also be ignoring all the side effects planes bring with them.
Would you not agree that if we only ever wanted "flight" for a specific use case that apparently only birds can do after evaluating what a plane cannot do, then planes are not capable of "flight"?
This is the very same problem with "thought" in terms of AI. We're finding it's inadequate for what we want the machine to do. Not only is it inadequate for our current use cases, and not only is it inadequate now, but it will continue to be inadequate until we further pin down what "thought" is and determine what lies beyond the Church-Turing thesis.
Relevant quote: "B. Jack Copeland states that it is an open empirical question whether there are actual deterministic physical processes that, in the long run, elude simulation by a Turing machine; furthermore, he states that it is an open empirical question whether any such processes are involved in the working of the human brain"
Yes, that's a problem of me not being a native english speaker.
"All x aren't y" may mean "not all x are y" in my tongue.
Not a single x is y is more what we would say in the previous case.
But in our case we would say there are x that aren't y.
If thinking is definable, it is wrong that all statements about it are unverifiable (i.e. there are statements about it that are verifiable.)
At the end of the day most people would agree that if something is able to solve a problem without a lookup table / memorisation that it used reasoning to reach the answer. You are really just splitting hairs here.
The difference between thinking and reasoning is that I can "think" that Elvis is still alive, Jewish space lasers are responsible for California wildfires, and Trump was re-elected president in 2020, but I cannot "reason" myself into those positions.
It ties into another aspect of these perennial threads, where it is somehow OK for humans to engage in deluded or hallucinatory thought, but when an AI model does it, it proves they don't "think."
>Equivalent statements could be made about how human brains are not magic, just biology - yet I think we still think.
They're not equivalent at all because the AI is by no means biological. "It's just maths" could maybe be applied to humans but this is backed entirely by supposition and would ultimately just be an assumption of its own conclusion - that human brains work on the same underlying principles as AI because it is assumed that they're based on the same underlying principles as AI.
Unless you're supposing something mystical or supernatural about how brains work, then yes, it is "just" math, there is nothing else it could be. All of the evidence we have shows it's an electrochemical network of neurons processing information. There's no evidence that suggests anything different, or even the need for anything different. There's no missing piece or deep mystery to it.
It's on those who want alternative explanations to demonstrate even the slightest need for them exists - there is no scientific evidence that exists which suggests the operation of brains as computers, as information processors, as substrate independent equivalents to Turing machines, are insufficient to any of the cognitive phenomena known across the entire domain of human knowledge.
We are brains in bone vats, connected to a wonderful and sophisticated sensorimotor platform, and our brains create the reality we experience by processing sensor data and constructing a simulation which we perceive as subjective experience.
The explanation we have is sufficient to the phenomenon. There's no need or benefit for searching for unnecessarily complicated alternative interpretations.
If you aren't satisfied with the explanation, it doesn't really matter - to quote one of Neil DeGrasse Tyson's best turns of phrase: "the universe is under no obligation to make sense to you"
If you can find evidence, any evidence whatsoever, and that evidence withstands scientific scrutiny, and it demands more than the explanation we currently have, then by all means, chase it down and find out more about how cognition works and expand our understanding of the universe. It simply doesn't look like we need anything more, in principle, to fully explain the nature of biological intelligence, and consciousness, and how brains work.
Mind as interdimensional radios, mystical souls and spirits, quantum tubules, none of that stuff has any basis in a ruthlessly rational and scientific review of the science of cognition.
That doesn't preclude souls and supernatural appearing phenomena or all manner of "other" things happening. There's simply no need to tie it in with cognition - neurotransmitters, biological networks, electrical activity, that's all you need.
>it doesn't really matter - to quote one of Neil DeGrasse Tyson's best turns of phrase: "the universe is under no obligation to make sense to you"
Right back at you, brochacho. I'm not the one making a positive claim here. You're the one who insists that it must work in a specific way because you can't conceive of any alternatives. I have never seen ANY evidence or study linking any existent AI or computer system to human cognition.
>There's no need or benefit for searching for unnecessarily complicated alternative interpretations.
Thanks, if it's alright with you I might borrow this argument next time somebody tries to tell me the world isn't flat.
>It simply doesn't look
That's one of those phrases you use when you're REALLY confident that you know what you're talking about.
> like we need anything more, in principle, to fully explain the nature of biological intelligence, and consciousness, and how brains work.
Please fully explain the nature of intelligence, consciousness, and how brains work.
>Mind as interdimensional radios, mystical souls and spirits, quantum tubules, none of that stuff has any basis in a ruthlessly rational and scientific review of the science of cognition.
well i definitely never said anything even remotely similar to that. If i didn't know any better i might call this argument a "hallucination".
Panpsychism is actually quite reasonable in part because it changes the questions you ask. Instead of “does it think” you need to ask “in what ways can it think, and in what ways is it constrained? What types of ‘experience/qualia’ can this system have, and what can’t it have?”
When you think in these terms, it becomes clear that LLMs can’t have certain types of experiences (eg see in color) but could have others.
A “weak” panpsychism approach would just stop at ruling out experience or qualia based on physical limitations. Yet I prefer the “strong” pansychist theory that whatever is not forbidden is required, which begins to get really interesting (would imply that for example an LLM actually experiences the interaction you have with it, in some way).
But parent didn't try to apply "it's just maths" to humans. He said one could just as easily say, as some do: "Humans are just biology, hence they're not magic". Our understanding of mathematics, including the maths of transformer models is limited, just as our understanding of biology. Some behaviors of these models have taken researches by surprise, and future surprises are not at all excluded. We don't know exactly how far they will evolve.
As for applying the word thinking to AI systems, it's already in common usage and this won't change. We don't have any other candidate words, and this one is the closest existing word for referencing a computational process which, one must admit, is in many ways (but definitely not in all ways) analogous to human thought.
Human brains and experiences seem to be constrained by the laws of quantum physics, which can be simulated to arbitrary fidelity on a computer. Nit sure where Godel’s incompleteness theory would even come in here…
how are we going to deduce/measure/know the initialization and rules for consciousness? do you see any systems as not encodable/simulatable by quantum?
I think you are asking whether consciousness might be a fundamentally different “thing” from physics and thus hard or impossible to simulate.
I think there is abundant evidence that the answer is ‘no’. The main reason is that consciousness doesn’t give you new physics, it follows the same rules and restrictions. It seems to be “part of” the standard natural universe, not something distinct.
Can you look at any arbitrary program and tell if it halts without running it indefinitely? If so, you should explain how and collect your Nobel. Telling everybody whether the Collatz conjecture is correct is a good warm up. If not, you can’t solve the halting program either. What does that have to do with consciousness though?
Having read “I Am a Strange Loop” I do not believe Hofstadter indicates that the existence of Gödel’s theorem precludes consciousness being realizable on a Turing machine. Rather if I recall correctly he points out that as a possible argument and then attempts to refute it.
On the other hand Penrose is a prominent believer that human’s ability to understand Gödel’s theorem indicates consciousness can’t be realized on a Turing machine but there’s far from universal agreement on that point.
per halting problem: any system capable of self reference has unprovable (un)truths, the system can not be complete and consistent. consciousness falls under this umbrella
I'll try and ask OG q more clearly: why would the brain, consciousness, be formalizable?
I think there's a yearn view nature as adhering to an underlying model, and a contrary view that consciousness is transcendental, and I lean towards the latter
> that human brains work on the same underlying principles as AI
That wasn't the assumption though, it was only that human brains work by some "non-magical" electro-chemical process which could be described as a mechanism, whether that mechanism followed the same principles of AI or not.
Straw man. The person who you're responding to talked about "equivalent statements" (emphasis added), whereas you appear to be talking about equivalent objects (AIs vs. brains), and pointing out the obvious flaw in this argument, that AIs aren't biology. The obvious flaw in the wrong argument, that is.
Indeed, people confidently assert as established fact things like "brains are bound by the laws of physics" and therefore "there can't be anything special" about them, so "consciousness is an illusion" and "the mind is a computer", all with absolute conviction but with very little understanding of what physics and maths really do and do not say about the universe. It's a quasi-religious faith in a thing not fully comprehended. I hope in the long run some humility in the face of reality will eventually be (re)learned.
If your position is that brains are not actually bound by the laws of physics -- that they operate on some other plane of existence unbound by any scientifically tested principle -- then it is not only your ideological opposites who have quasi-religious faith in a thing not fully comprehended.
My "position" isn't remotely that. The problem with "brains are bound by the laws of physics" isn't that there's something special about brains. It's that physics doesn't consist of "laws" that things are "bound" by. It consists of theories that attempt to describe.
These theories are enormously successful, but they are also known to be variously incomplete, inconsistent, non-deterministic, philosophically problematic, open to multiple interpretations and only partially understood in their implications, with links between descriptions of things at different scales a particularly challenging and little understood topic. The more you learn about physics (and while I'm no physicist, I have a degree in the subject and have learned a great deal more since) the more you understand the limits of what we know.
Anybody who thinks there's no mystery to physics just doesn't know much about it. Anybody who confidently
asserts as fact things like "the brain consists of protons, neutrons and electrons so it's impossible for it to do anything a computer can't do" is deducing things from their own ignorance.
This. People do not understand the implications of the most basic facts of modern science. Gravitation is instantaneous action at a distance via an "occult" force (to quote Newton's contemporaries).
Lot's of assumptions about humanity and how unique we are constantly get paraded in this conversation. Ironically, the people who tout those perspectives are the least likely to understand why we're really not all that "special" from a very factual and academic perspective.
You'd think it would unlock certain concepts for this class of people, but ironically, they seem unable to digest the information and update their context.
A large number of adults I encounter are functionally illiterate, including business people in very high up positions. They are also almost 100% MATHEMATICALLY illiterate, not only unable to solve basic algebra and geometry problems, but completely unable to reason about statistical and probabilistic situations encountered in every day life. This is why gambling is so popular and why people are constantly fooled by politicians. It's bad enough to be without words in the modern world, but being without numbers makes you vulnerable to all manner of manipulations.
Gambling exists more because of people dopamine systems than math...though I get the overall drift. People are fooled by politicians because ?? Also not really math related I think.
I have yet to hear any plausible definition of "thought" that convincingly places LLMs and brains on opposite sides of it without being obviously contrived for that purpose.
A college level approach could look at the line between Math/Science/Physics and Philosophy. One thing from the article that stood out to me was that the introduction to their approach started with a problem about classifying a traffic light. Is it red or green?
But the accompanying XY plot showed samples that overlapped or at least were ambiguous. I immediately lost a lot of my interest in their approach, because traffic lights by design are very clearly red, or green. There aren't mauve or taupe lights that the local populace laughs at and says, "yes, that's mostly red."
I like the idea of studying math by using ML examples. I'm guessing this is a first step and future education will have better examples to learn from.
> traffic lights by design are very clearly red, or green
I suspect you feel this because you are observing the output of a very sophisticated image processing pipeline in your own head. When you are dealing with raw matrixes of rgb values it all becomes a lot more fuzzy. Especially when you encounter different illuminations, exposures and the cropping of the traffic light has noise on it. Not saying it is some intractably hard machine vision problem, because it is not. But there is some variety and fuzzyness there in the raw sensor measurements.
We observe through our senses geometric relationships.
Syntax is exactly that; letters, sentences, paragraphs organized in spatial/geometric relationships.
At best thought is recreation of neural networks in the brain which only exist as spatial relationships.
Our senses operate on spatial relationships; enough light to work by, and food relative to stomach to satisfy our biological impetus to survive (which is spatial relationships of biochemistry).
The idea of "thought" as anything but biology makes little sense to me then as a root source is clearly observable. Humanity, roughly, repeats the same social story. All that thought does not seem to be all that useful as we end up in the same place; the majority as serfs of aristocracy.
Personally would prefer less "thought" role-play and more people taking the load of the labor they exploit to enable them to sit and "think".
We really don't know how consciousness works. The popular theories that it's emergent might be proven correct, or might be proven to be like the idea that phlogiston built up in a vacuum, putting out flames.
That's where these threads always end up. Someone asserts, almost violently, that AI does not and/or cannot "think." When asked how to falsify their assertion, perhaps by explaining what exactly is unique about the human brain that cannot and/or will not be possible to emulate, that's the last anyone ever hears from them. At least until the next "AI can't think" story gets posted.
The same arguments that appeared in 2015 inevitably get trotted out, almost verbatim, ten years later. It would be amusing on other sites, but it's just pathetic here.
Consider that you might have become polarized yourself. I often encounter good arguments against current AI systems emulating all essential aspects of human thinking. For example, the fact that they can't learn from few examples, that they can't perform simple mathematical operations without access to external help (via tool calling) or that they have to expend so much more energy to do their magic (and yes, to me they are a bit magical), which makes some wonder if what these models do is a form of refined brute-force search, rather than ideating.
Personally, I'm ok with reusing the word "thinking", but there are dogmatic stances on both sides. For example, lots of people decreeing that biology in the end can't but reduce to maths, since "what else could it be". The truth is we don't actually know if it is possible, for any conceivable computational system, to emulate all essential aspects of human thought. There are good arguments for this (in)possibility, like those presented by Roger Penrose in "the Emperor's new Mind" and "Shadows of the Mind".
For example, the fact that they can't learn from few examples
For one thing, yes, they can, obviously [1] -- when's the last time you checked? -- and for another, there are plenty of humans who seemingly cannot.
The only real difference is that with an LLM, when the context is lost, so is the learning. That will obviously need to be addressed at some point.
that they can't perform simple mathematical operations without access to external help (via tool calling)
But yet you are fine with humans requiring a calculator to perform similar tasks? Many humans are worse at basic arithmetic than an unaided transformer network. And, tellingly, we make the same kinds of errors.
or that they have to expend so much more energy to do their magic (and yes, to me they are a bit magical), which makes some wonder if what these models do is a form of refined brute-force search, rather than ideating.
Well, of course, all they are doing is searching and curve-fitting. To me, the magical thing is that they have shown us, more or less undeniably (Penrose notwithstanding), that that is all we do. Questions that have been asked for thousands of years have now been answered: there's nothing special about the human brain, except for the ability to form, consolidate, consult, and revise long-term memories.
That's post-training. The complaint I'm referring to is to the huge amounts of data (end energy) required during training - which is also a form of learning, after all. Sure, there are counter-arguments, for example pointing to the huge amount of non-textual data a child ingests, but these counter-arguments are not waterproof themselves (for example, one can point out that we are discussing text-only tasks). The discussion can go on and on, my point was only that cogent arguments are indeed often presented, which you were denying above.
> there are plenty of humans who seemingly cannot
This particular defense of LLMs has always puzzled me. By this measure, simply because there are sufficiently impaired humans, AGI has already been achieved many decades ago.
> But yet you are fine with humans requiring a calculator to perform similar tasks
I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator.
> To me, the magical thing is that they have shown us, more or less undeniably (Penrose notwithstanding), that that is all we do.
Or, to put it more tersely, they have shown you that that is all we do. Penrose, myself, and lots of others don't see it quite like that. (Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. ;) To me what LLMs do is approximate one aspect of our minds. But I have a strong hunch that the rabbit hole goes much deeper, your assessment notwithstanding.
No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model."
I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator.
So am I. See, for example, Karpathy's discussion of native computation: https://youtu.be/7xTGNNLPyMI?si=Gckcmp2Sby4SlKje&t=6416 (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers.
You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week.
Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly.
Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
Yes, it is. You seem to have misunderstood what I wrote. The critique I was pointing to is of the amount of examples and energy needed during model training, which is what the "learning" in "machine learning" actually refers to. The paper uses GPT-3 which had already absorbed all that data and electricity. And the "learning" the paper talks about is arguably not real learning, since none of the acquired skills persists beyond the end of the session.
> So am I.
This is easy to settle. Go check any frontier model and see how far they get with multiplying numbers with tool calling disabled.
> Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
Neither am I resting, nor are there much laurels to rest on, at least compared to someone like Penrose. As for him, give the man a break, he's 94 years old and still sharp as a tack and intellectually productive. You're the one who's resting, imagining you've settled a question which is very much still open. Certainty is certainly intoxicating, so I understand where you're coming from, but claiming anyone who doubts computationalism is not bringing any arguments to the table is patently absurd.
Usually it is the work of the one claiming something to prove it.
So if you believe that AI does "think" you are expected to show me that it really does.
Claiming it "thinks - prove otherwise" is just bad form and also opens the discussion up for moving the goalposts just as you did with your brain emulation statement. Or you could just not accept any argument made or circumvent it by stating the one trying to disprove your assertion got the definition wrong.
There are countless ways to start a bad faith argument using this methodology, hence: Define property -> prove property.
Conversely, if the one asserting something doesn't want to define it there is no useful conversation to be had. (as in: AI doesn't think - I won't tell you what I mean by think)
PS: Asking someone to falsify their own assertion doesn't seem a good strategy here.
PPS: Even if everything about the human brain can be emulated, that does not constitute progress for your argument, since now you'd have to assert that AI emulates the human brain perfectly before it is complete. There is no direct connection between "This AI does not think" to "The human brain can be fully emulated". Also the difference between "does not" and "can not" is big enough here that mangling them together is inappropriate.
So if you believe that AI does "think" you are expected to show me that it really does.
A lot of people seemingly haven't updated their priors after some of the more interesting results published lately, such as the performance of Google's and OpenAI's models at the 2025 Math Olympiad. Would you say that includes yourself?
If so, what do the models still have to do in order to establish that they are capable of all major forms of reasoning, and under what conditions will you accept such proof?
It definietly includes myself, I don't have the interest to stay updated here.
For that matter I have no opinion on if AI does think or not, I simply don't care.
Therefore I also really can't answer your question in what more a model has to do to establish that they are thinking (does being able to use all major forms of reasoning constitute the capability of thought to you?).
I can say however, that any such proof would have to be on a case-by-case basis given my current understanding on AI is designed.
Well first of all I never claimed that I was capable of thinking (smirk).
We also haven't agreed on a definition of "thinking" yet, so as you can read in my previous comment, there is no meaningful conversation to be had.
I also don't understand how your oddly aggresive phrasing adds to the conversation,
but if it helps you: my rights and protections do not depend on whether I'm able to prove to you that I am thinking.
(It also derails the conversation for what it's worth - it's a good strategy in the debating club, but these are about winning or loosing and not about fostering and obtaining knowledge)
Whatever you meant to say with "Sometimes, because of the consequences of otherwise, the order gets reversed" eludes me as well.
If I say I'm innocent, you don't say I have to prove it. Some facts are presumed to be true without burden of evidence because otherwise it could cause great harm.
So we don't require, say, minorities or animals to prove they have souls, we just inherently assume they do and make laws around protecting them.
Thank you for the clarification.
If you expect me to justify an action depending on you being innocent, then I actually do need you to prove it.
I wouldn't let you sleep in my room assuming you're innocent - or in your words: because of the consequences of otherwise.
It feels like you're moving the goalposts here: I don't want to justify an action based on something, i just want to know if something has a specific property.
With regards to the topic: Does AI think?
I don't know, but I also don't want to act upon knowing if it does (or doesn't for that matter).
In other words, I don't care.
The answer could go either way, but I'd rather say that I don't know (especially since "thinking" is not defined).
That means that I can assume both and consider the consequences using some heuristic to decide which assumption is better given the action I want to justify doing or not doing.
If you want me to believe an AI thinks, you have to prove it, if you want to justify an action you may assume whatever you deem most likely.
And if you want to know if an AI thinks, then you literally can't assume it does; simple as that.
Someone asserts, almost religiously, that LLMs do and/or can "think." When asked how to falsify their assertion, perhaps by explaining what exactly is "thinking" in the human brain that can and/or will be possible to emulate...
Err, no, that’s not what’s happening. Nobody, at least in this thread (and most others like it I’ve seen), is confidently claiming LLMs can think.
There are people confidently claiming they can’t and then other people expressing skepticism at their confidence and/or trying to get them to nail down what they mean.
Or they just point to the turing test which was the defacto standard test for something so nebulous. And behold: LLM can pass the turing test. So they think. Can you come up with something better (than the turing test)?
But the Turing test (which I concede, LLMs do pass) doesn't test if some system is thinking; it tests if the system can convince an unbiased observer that it is thinking. I cannot come up with a better "is this thing thinking" test, but that doesn't mean that such a test can't exist; I'm sure there are much smarter people then me trying to solve this problem.
When asked how to falsify their assertion, perhaps by explaining what exactly is "thinking" in the human brain that can and/or will be possible to emulate...
... someone else points out that the same models that can't "think" are somehow turning in gold-level performance at international math and programming competitions, making Fields Medalists sit up and take notice, winning art competitions, composing music indistinguishable from human output, and making entire subreddits fail the Turing test.
> That's kind of a big difference, wouldn't you say?
To their utility.
Not sure if it matters on the question "thinking?"; even if for the debaters "thinking" requires consciousness/qualia (and that varies), there's nothing more than guesses as to where that emerges from.
For my original earlier reply, the main subtext would be: "Your complaint is ridiculously biased."
For the later reply about chess, perhaps: "You're asserting that tricking, amazing, or beating a human is a reliable sign of human-like intelligence. We already know that is untrue from decades of past experience."
You're asserting that tricking, amazing, or beating a human is a reliable sign of human-like intelligence.
I don't know who's asserting that (other than Alan Turing, I guess); certainly not me. Humans are, if anything, easier to fool than our current crude AI models are. Heck, ELIZA was enough to fool non-specialist humans.
In any case, nobody was "tricked" at the IMO. What happened there required legitimate reasoning abilities. The burden of proof falls decisively on those who assert otherwise.
I feel like these conversations really miss the mark: whether an LLM thinks or not is not a relevant question. It is a bit like asking “what color is an Xray?” or “what does the number 7 taste like?”
The reason I say this is because an LLM is not a complete self-contained thing if you want to compare it to a human being. It is a building block. Your brain thinks. Your prefrontal cortex however is not a complete system and if you somehow managed to extract it and wire it up to a serial terminal I suspect you’d be pretty disappointed in what it would be capable of on its own.
I want to be clear that I am not making an argument that once we hook up sensory inputs and motion outputs as well as motivations, fears, anxieties, desires, pain and pleasure centers, memory systems, sense of time, balance, fatigue, etc. to an LLM that we would get a thinking feeling conscious being. I suspect it would take something more sophisticated than an LLM. But my point is that even if an LLM was that building block, I don’t think the question of whether it is capable of thought is the right question.
Given the context of the article, why would you ignore that? It's important to discuss the underlying technology in the context in which it's being sold to the public at large.
Why do people call is "Artificial Intelligence" when it could be called "Statistical Model for Choosing Data"?
"Intelligence" implies "thinking" for most people, just as "Learning" in machine learning implies "understanding" for most people. The algorithms created neither 'think' nor 'understand' and until you understand that, it may be difficult to accurately judge the value of the results produced by these systems.
Actually I think the name is apt. It's artificial. It's like how an "artificial banana" isn't actually a banana. It doesn't have to be real thinking or real learning, it just has to look intelligent (which it does).
The term was coined in 1955 to describe "learning or any other feature of intelligence" simulated by a machine [1]. The same proposal does list using natural language as one of the aspects of "the artificial intelligence problem"
It's not a perfect term, but we have been using it for seven full decades to include all of machine learning and plenty of things even less intelligent
Same way I feel about 'Military Intelligence' :-). Both of those phrases use the 'information gathering and analysis' definition of intelligence rather than the 'thinking' definition.
You should take your complaints to OpenAI, who constantly write like LLMs think in the exact same sense as humans; here a random example:
> Large language models (LLMs) can be dishonest when reporting on their actions and beliefs -- for example, they may overstate their confidence in factual claims or cover up evidence of covert actions
They have a product to sell based on the idea AGI is right around the corner. You can’t trust Sam Altman as far as you can throw him.
Still, the sales pitch has worked to unlock huge liquidity for him so there’s that.
Still making predictions is a big part of what brains do though not the only thing. Someone wise said that LLM intelligence is a new kind of intelligence, like how animal intelligence is different from ours but is still intelligence but needs to be characterized to understand differences.
> Someone wise said that LLM intelligence is a new kind of intelligence
So long as you accept the slide ruler as a "new kind of intelligence" everything will probably work out fine, it's the Altmannian insistence that only the LLM is of the new kind that is silly.
This is the underlying problem behind syncophantcy.
I saw a YouTube video about a investigative youtuber Eddy Burback who very easily convinced chat4 that he should cut off all contact with friends and family, move to a cabin in the desert, eat baby food, wrap himself in alfoil, etc just feeding his own (faked) mistakes and delusions. "What you are doing is important, trust your instincts".
Wven if AI could hypothetically be 100x as smart as a human under the hood, it still doesn't care. It doesn't do what it thinks it should, it doesn't do what it needs to do, it does what we train it to.
We train in humanities weaknesses and follies. AI can hypothetically exceed humanity in some respects, but in other respects it is a very hard to control power tool.
AI is optimised, and optimised functions always "hack" the evaluation function. In the case of AI, the evaluation function includes human flaws. AI is trained to tell us what we want to hear.
Elon Musk sees the problem, but his solution is to try to make it think more like him, and even if that succeeds it just magnifies his own weaknesses.
Has anyone read the book criticising Ray Dalio? He is a very successful hedge fund manager, who decided that he could solve the problem of finding a replacement by psychology evaluation and training people to think like him. But even his smartest employees didn't think like him, they just (reading between the lines) gamed his system. Their incentives weren't his incentives - he could demand radical honesty and integrity but that doesn't work so well when he would (of course) reward the people who agreed with him, rather than the people who would tell him he was screwing up. His organisation (apparently) became a bunch of even more radical syncopants due to his efforts to weed out syncophantcy.
The part that's most infuriating is that we don't have to speculate at all. Any discussion or philosophizing beyond the literal computer science is simply misinformation.
There's absolutely no similarity between what computer hardware does and what a brain does. People will twist and stretch things and tickle the imagination of the naive layperson and that's just wrong. We seriously have to cut this out already.
Anthropomorphizing is dangerous even for other topics, and long understood to be before computers came around. Why do we allow this?
The way we talk about computer science today sounds about as ridiculous as invoking magic or deities to explain what we now consider high school physics or chemistry. I am aware that the future usually sees the past as primitive, but why can't we try to seem less dumb at least this time around?
> There's absolutely no similarity between what computer hardware does and what a brain does.
But at very least there's also no similarity between what computer hardware does and what even the simplest of LLMs do. They don't run on eg. x86_64 , else qemu would be sufficient for inferencing.
You can replicate all calculations done by LLMs with pen and paper. It would take ages to calculate anything, but it's possible. I don't think that pen and paper will ever "think", regardless of how complex the calculations involved are.
And the counter argument is also exactly the same. Imagine you take one neuron from a brain and replace it with an artificial piece of electronics (e.g. some transistors) that only generates specific outputs based on inputs, exactly like the neuron does. Now replace another neuron. And another. Eventually, you will have the entire brain replaced with a huge set of fundamentally super simple transistors. I.e. a computer. If you believe that consciousness or the ability to think disappears somewhere during this process, then you are essentially believing in some religious meta-physics or soul-like component in our brains that can not be measured. But if it can not be measured, it fundamentally can not affect you in any way. So it doesn't matter for the experiment in the end, because the outcome would be exactly the same. The only reason you might think that you are conscious and the computer is not is because you believe so. But to an outsider observer, belief is all it is. Basically religion.
It seems like the brain "just" being a giant number of neurons is an assumption. As I understand it's still an area of active research, for example the role of glial cells. The complete function may or may not be pen and paper-able.
There are indeed many people trying to justify this magical thinking by seeking something, anything in the brain that is out of the ordinary. They've been unsuccessful so far.
Penrose comes to mind, he will die on the hill that the brain involves quantum computations somehow, to explain his dualist position of "the soul being the entity responsible for deciding how the quantum states within the brain collapse, hence somehow controlling the body" (I am grossly simplifying). But even if that was the case, if the brain did involve quantum computations, those are still, well, computable. They just involve some amount of randomness, but so what? To continue with grandparent's experiment, you'd have to replace biological neurons with tiny quantum computer neurons instead, but the gist is the same.
You wouldn't even need quantum computer neurons. We can simulate quantum nature on normal circuits, albeit not very efficiently. But for the experiment this wouldn't matter. The only important thing would be that you can measure it, which in turn would allow you to replicate it in some non-human circuit. And if you fundamentally can't measure this aspect for some weird reason, you will once again reach the same conclusion as above.
You can simulate it, but you usually use PRNG to decide how your simulated wave function "collapses". So in the spirit of the original thought experiment, I felt it more adequate to replace the quantum part (if it even exists) by another actually quantum part. But indeed, using fake quantum shouldn't change a thing.
It could well be the case that the brain can be simulated, but presently we don't know exactly what variables/components must be simulated. Does ongoing neuroplasticity for example need to be a component of simulation? Is there some as of yet unknown causal mechanisms or interactions that may be essential?
None of those examples cannot be done on pen and paper or otherwise simulated with a different medium, though.
AFAICT, your comment above would need some mechanism that is physically impossible and incalculable to make the argument, and then somehow have that happen in a human brain despite being physically impossible and incalculable.
> component in our brains that can not be measured.
"Can not be measured", probably not. "We don't know how to measure", almost certainly.
I am capable of belief, and I've seen no evidence that the computer is. It's also possible that I'm the only person that is conscious. It's even possible that you are!
That appears to be your own assumptions coming into play.
Everything I've seen says "LLMs cannot think like brains" is not dependent on an argument that "no computer can think like a brain", but rather on an understanding of just what LLMs are—and what they are not.
I don’t understand why people say the Chinese Room thing would prove LLMs don’t think, to me it’s obvious that the person doesn’t understand Chinese but the process does, similarly the CPU itself doesn’t understand the concepts an LLM can work with but the LLM itself does, or a neuron doesn’t understand concepts but the entire structure of your brain does
The concept of understanding emerges on a higher level from the way the neurons (biological or virtual) are connected, or the way the instructions being followed by the human in the Chinese room process the information
But really this is a philosophical/definitional thing about what you call “thinking”
Edit: I see my take on this is listed on the page as the “System reply”
If 100 top-notch philosophers disagree with you, that means you get 100 citations from top-notch philosophers. :-P
Check out eg Dennett.... or ... his opionions about Searle; Have fun with eg... this:
"By Searle’s own count, there are over a hundred published attacks on it. He can count them, but I guess he can’t read them, for in all those years he has never to my knowledge responded in detail to the dozens of devastating criticisms they contain;"
I don't see the relevance of that argument (which other responders to your post have pointed out as Searle's Chinese Room argument). The pen and paper are of course not doing any thinking, but then the pen isn't doing any writing on its own, either. It's the system of pen + paper + human that's doing the thinking.
The idea of my argument is that I notice that people project some "ethereal" properties over computations that happen in the... computer. Probably because electricity is involved, making things show up as "magic" from our point of view, making it easier to project consciousness or thinking onto the device. The cloud makes that even more abstract. But if you are aware that the transistors are just a medium that replicates what we already did for ages with knots, fingers, and paint, it gets easier to see them as plain objects.
Even the resulting artifacts that the machine produces are only something meaningful from our point of view, because you need prior knowledge to read the output signals. So yeah, those devices end up being an extension of ourselves.
Your view is missing the forest for the trees. You see individual objects but miss the aggregate whole. You have a hard time conceiving of how exotic computers can be conscious because we are scale chauvinists by design. Our minds engage with the world on certain time and length scales, and so we naturally conceptualize our world based on entities that exist on those scales. But computing is necessarily scale independent. It doesn't matter to the computation if it is running on some 100GHz substrate or .0001Hz. It doesn't matter if its running on a CPU chip the size of a quarter or spread out over the entire planet. Computation is about how information is transformed in semantically meaningful ways. Scale just doesn't matter.
If you were a mind supervening on the behavior of some massive time/space scale computer, how would you know? How could you tell the difference between running on a human making marks with pen and paper and running on a modern CPU? Your experience updates based on information transformations, not based on how fast the fundamental substrate is changing. When your conscious experience changes, that means your current state is substantially different from your prior state and you can recognize this difference. Our human-scale chauvinism gets in the way of properly imagining this. A mind running on a CPU or a large collection of human computers is equally plausible.
A common question people like to ask is "where is the consciousness" in such a system. This is an important question if only because it highlights the futility of such questions. Where is Microsoft Word when it is running on my computer? How can you draw a boundary around a computation when there are a multitude of essential and non-essential parts of the system that work together to construct the relevant causal dynamic. It's just not a well-defined question. There is no one place where Microsoft Word occurs nor is there any one place where consciousness occurs in a system. Is state being properly recorded and correctly leveraged to compute the next state? The consciousness is in this process.
If you put a droplet of water in a warm bowl every 12 hours, the bowl will remain empty as the water will evaporate. That does not mean that if you put a trillion droplets in every twelve hours it will still remain empty.
The point I was trying to make was that the time you use to perform the calculation may change whether there is an "experience" on behalf of the calculation. Without specifying the basis if subjectivity, you can't rule anything out as far as what matters and what doesn't. Maybe the speed or locality with which the calculations happen matters. Like the water drops: given the same amount of time, eventually all the water will evaporate in either case leading to the same end state, but the the intermediate states are very different.
You can replicate the entire universe with pen and paper (or a bunch of rocks). It would take an unimaginably long time, and we haven't discovered all the calculations you'd need to do yet, but presumably they exist and this could be done.
Does that actually make a universe? I don't know!
The comic is meant to be a joke, I think, but I find myself thinking about it all the time!!!
Even worse, as we are part of the universe, we would need to simulate ourselves and the very simulation that we are creating. You would also need to replicate the simulation of the simulation, leading to an eternal loop that would demand infinite matter and time (and would still not be enough!). Probably, you can't simulate something while being part of it.
It doesn’t need to be our universe, just a universe.
The question is, are the people in the simulated universe real people? Do they think and feel like we do—are they conscious? Either answer seems like it can’t possibly be right!
> You can simulate a human brain on pen and paper too.
That's an assumption, though. A plausible assumption, but still an assumption.
We know you can execute an LLM on pen and paper, because people built them and they're understood well enough that we could list the calculations you'd need to do. We don't know enough about the human brain to create a similar list, so I don't think you can reasonably make a stronger statement than "you could probably simulate..." without getting ahead of yourself.
I can make a claim much stronger than "you could probably" The counterclaim here is that the brain may not obey physical laws that can be described by mathematics. This is a "5G causes covid" level claim. The overwhelming burden of proof is on you.
There are some quantum effects in the brain (for some people, that's a possible source of consciousness).
We can simulate quantum effects, but here comes the tricky part: even if our simulation matches the probability, say 70/30 of something happening, what guarantees that our simulation would take the same path as the object being simulated?
We don't have to match the quantum state since the brain still produces an valid output regardless of what each random quantum probability ended up as. And we can include random entropy in a LLM too.
This is just non-determinism. Not only can't your simulation reproduce the exact output, but neither can your brain reproduce its own previous state. This doesn't mean it's a fundamentally different system.
Consider for example Orch OR theory. If it or something like it were to be accurate, the brain would not "obey physical laws that can be described by mathematics".
Orch OR is probably wrong, but the broader point is that we still don’t know which physical processes are necessary for cognition. Until we do, claims of definitive brain simulability are premature.
This is basically the Church-Turing thesis and one of the motivations of using tape(paper) and an arbitrary alphabet in the Turing machine model.
It's been kinda discussed to oblivion in the last century, interesting that it seems people don't realize the "existing literature" and repeat the same arguments (not saying anyone is wrong).
The simulation isn't an operating brain. It's a description of one. What it "means" is imposed by us, what it actually is, is a shitload of graphite marks on paper or relays flipping around or rocks on sand or (pick your medium).
An arbitrarily-perfect simulation of a burning candle will never, ever melt wax.
An LLM is always a description. An LLM operating on a computer is identical to a description of it operating on paper (if much faster).
What makes the simulation we live in special compared to the simulation of a burning candle that you or I might be running?
That simulated candle is perfectly melting wax in its own simulation. Duh, it won't melt any in ours, because our arbitrary notions of "real" wax are disconnected between the two simulatons.
If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
>If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
I can smell a "real" candle, a "real" candle can burn my hand. The term real here is just picking out a conceptual schema where its objects can feature as relata of the same laws, like a causal compatibility class defined by a shared causal scope. But this isn't unique to the question of real vs simulated. There are causal scopes all over the place. Subatomic particles are a scope. I, as a particular collection of atoms, am not causally compatible with individual electrons and neutrons. Different conceptual levels have their own causal scopes and their own laws (derivative of more fundamental laws) that determine how these aggregates behave. Real (as distinct from simulated) just identifies causal scopes that are derivative of our privileged scope.
Consciousness is not like the candle because everyone's consciousness is its own unique causal scope. There are psychological laws that determine how we process and respond to information. But each of our minds are causally isolated from one another. We can only know of each other's consciousness by judging behavior. There's nothing privileged about a biological substrate when it comes to determining "real" consciousness.
That's a fair reading but not what I was going for. I'm trying to argue for the irrelevance of causal scope when it comes to determining realness for consciousness. We are right to privilege non-virtual existence when it comes to things whose essential nature is to interact with our physical selves. But since no other consciousness directly physically interacts with ours, it being "real" (as in physically grounded in a compatible causal scope) is not an essential part of its existence.
Determining what is real by judging causal scope is generally successful but it misleads in the case of consciousness.
> If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
A candle in Canada can't melt wax in Mexico, and a real candle can't melt simulated wax. If you want to differentiate two things along one axis, you can't just point out differences that may or may not have any effect on that axis. You have to establish a causal link before the differences have any meaning. To my knowledge, intelligence/consciousness/experience doesn't have a causal link with anything.
We know our brains cause consciousness the way we knew in 1500 that being on a boat for too long causes scurvy. Maybe the boat and the ocean matter, or maybe they don't.
I think the core trouble is that it's rather difficult to simulate anything at all without requiring a human in the loop before it "works". The simulation isn't anything (well, it's something, but it's definitely not what it's simulating) until we impose that meaning on it. (We could, of course, levy a similar accusation at reality, but folks tend to avoid that because it gets uselessly solipsistic in a hurry)
A simulation of a tree growing (say) is a lot more like the idea of love than it is... a real tree growing. Making the simulation more accurate changes that not a bit.
I believe that the important part of a brain is the computation it's carrying out. I would call this computation thinking and say it's responsible for consciousness. I think we agree that this computation would be identical if it were simulated on a computer or paper.
If you pushed me on what exactly it means for a computation to physically happen and create consciousness, I would have to move to statements I'd call dubious conjectures rather than beliefs - your points in other threads about relying on interpretation have made me think more carefully about this.
Thanks for stating your views clearly. I have some questions to try and understand them better:
Would you say you're sure that you aren't in a simulation while acknowledging that a simulated version of you would say the same?
What do you think happens to someone whose neurons get replaced by small computers one by one (if you're happy to assume for the sake of argument that such a thing is possible without changing the person's behavior)?
It seems to me that the distinction becomes irrelevant as soon as you connect inputs and outputs to the real world. You wouldn't say that a 737 autopilot can never, ever fly a real jet and yet it behaves exactly the same whether it's up in the sky or hooked up to recorded/simulated signals on a test bench.
The brain follows the laws of physics. The laws of physics can be closely approximated by mathematical models. Thus, the brain can be closely approximated by mathematical models.
It’s not that open. We can simulate smaller system of neurons just fine, we can simulate chemistry. There might be something beyond that in our brains for some reason, but it sees doubtful right now
Our brains actually do something, may be the difference. They're a thing happening, not a description of a thing happening.
Whatever that something that it actually does in the real, physical world is produces the cogito in cogito, ergo sum and I doubt you can get it just by describing what all the subatomic particles are doing, any more than a computer or pen-and-paper simulated hurricane can knock your house down, no matter how perfectly simulated.
Doing something merely requires I/O. Brains wouldn't be doing much without that. A sufficiently accurate simulation of a fundamentally computational process is really just the same process.
Why are the electric currents moving in a GPU any less of a "thing happening" than the firing of the neurons in your brain? What you are describing here is a claim that the brain is fundamentally supernatural.
Thinking that making scribbles that we interpret(!!!) as perfectly describing a functioning consciousness and its operation, on a huge stack of paper, would manifest consciousness in any way whatsoever (hell, let's say we make it an automated flip-book, too, so it "does something"), but if you made the scribbles slightly different it wouldn't work(!?!? why, exactly, not ?!?!), is what's fundamentally supernatural. It's straight-up Bronze Age religion kinds of stuff (which fits—the tech elite is full of that kind of shit, like mummification—er, I mean—"cryogenic preservation", millenarian cults er, I mean The Singularity, et c)
Of course a GPU involves things happening. No amount of using it to describe a brain operating gets you an operating brain, though. It's not doing what a brain does. It's describing it.
(I think this is actually all somewhat tangential to whether LLMs "can think" or whatever, though—but the "well of course they might think because if we could perfectly describe an operating brain, that would also be thinking" line of argument often comes up, and I think it's about as wrong-headed as a thing can possibly be, a kind of deep "confusing the map for the territory" error; see also comments floating around this thread offhandedly claiming that the brain "is just physics"—like, what? That's the cart leading the horse! No! Dead wrong!)
You're arguing for the existence of a soul, for dualism. Nothing wrong with that, except we have never been able to measure it, and have never had to use it to explain any phenomenon of the brain's working. The brain follows the rules of physics, like any other objects of the material world.
A pen and paper simulation of a brain would also be "a thing happening" as you put it. You have to explain what is the magical ingredient that makes the brain's computations impossible to replicate.
You could connect your brain simulation to an actual body, and you'd be unable to tell the difference with a regular human, unless you crack it open.
You're arguing against Functionalism [0], of which I'd encourage you to at least read the Wikipedia page. Why would doing the brain's computations on pen and paper rather than on wetware lead to different outcomes? And how?
Connect your pen and paper operator to a brainless human body, and you got something indistinguishable from a regular alive human.
Wouldn't 'thinking' need to be updating the model of reality (LLM is not yet that, just words) - at every step doing again all that extensive calculations as when/to creating/approximating that/better model (learning) ?
Expecting machines to think is.. like magical thinking (but they are good at calculations indeed).
I wish we didn't use the word intelligence in context of LLMs - shortly there is Essence and the rest.. is only slope - into all possible combinations of Markov Chains - may they have sense or not I don't see how part of some calculation could recognize it, or that to be possible from inside (of calculation, that doesn't even consider that).
Aside of artificial knowledge (out of senses, experience, context lengths.. - confabulating but not knowing that), I wish to see an intelligent knowledge - made in kind of semantic way - allowed to expand using not yet obvious (but existing - not random) connections. I wouldn't expect it to think
(humans think, digitals calculate). But I would expect it to have a tendency to be coming closer (not further) in reflecting/modeling reality and expanding implications.
Thinking is different than forming long term memories.
An LLM could be thinking in one of two ways. Either between adding each individual token, or collectively across multiple tokens. At the individual token level the physical mechanism doesn’t seem to fit the definition being essentially reflexive action, but across multiple tokens that’s a little more questionable especially as multiple approaches are used.
An LLM ..is calculated ..from language (or from things being said by humans before being true or not). It's not some antropomorfic process what using the word thinking would suggest (to sell well).
> across multiple tokens
- but how many ? how many of them happen in sole person life ? How many in some calculation ? Does it matter, if a calculation doesn't reflect it but stay all the same ? (conversation with.. a radio - would it have any sense ?)
The general public have no issue saying a computer is thinking when you’re sitting there waiting for it to calculate a route or doing a similar process like selecting a chess move.
The connotation is simply an internal process of indeterminate length rather than one of reflexive length. So they don’t apply it when a GPU is slinging out 120 FPS in a first person shooter.
That's right when saying selecting not calculating a chess move - assuming you are outside of Plato's cave (Popper).
But now, I see this: the truth is static and non-profit, but calculating something can be sold again and again, if you have a hammer (processing) everything looks like a nail,
to sell well the word thinking had to be used instead of excuse for every time results being different (like the shadows)
- then, we can have only things that let someone else keep making profits: JS, LLM, whatever.. (just not.. "XSLT" alike).
.. and confront about Prolog or else in recent years - likes: "intended benefit requires an unreasonably (or impossibly?) smart compiler" (https://news.ycombinator.com/item?id=14441045) - isn't quite similar to LLMs, for that, requiring.. impossibly smart users ?? (there were few - assuming they got what they wanted . not peanuts)
LLMs don't really think, they emulate their training data. Which has a lot of examples of humans walking through problems to arrive at an answer. So naturally, if we prompt an LLM to do the same, it will emulate those examples (which tend to be more correct).
LLMs are BAD at evaluating earlier thinking errors, precisely because there's not copious examples of text where humans thinking through a problem, screwing up, going back, correcting their earlier statement, and continuing. (a good example catches these and corrects them)
> Secondary school maths showing that AI systems don’t think
And the article contains the quotes:
> the team wants to tackle a major and common misconception: that students think that ANN systems learn, recognise, see, and understand, when really it’s all just maths.
> The team is taking very complex ideas and reducing them to such an extent that we can use secondary classroom maths to show that AI is not magic and AI systems do not think.
In every reasonable use case for LLMs verifying the answer is trivial. Does the code do what I wanted it to? Does it link to a source that corroborates the response?
If you're asking for things you can't easily verify you're barking up the wrong tree.
It's not always the case, but often verifying an answer is far easier than coming up with the answer in the first place. That's precisely the principle behind the RSA algorithm for cryptography.
Yes. But it's offtopic because the presence of a provocative word 'thinking' in the title led to a lot of generic tangents that don't engage with anything interesting in the article, and mostly just express people's pre-existing associations about a controversial point.
Trying to avoid this kind of thing is why the guidelines say things like:
"Eschew flamebait. Avoid generic tangents."
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
A lot of the drama here is due to the ambiguity of what the word 'think' is supposed to mean. One camp associates 'thinking' to consciousness, another does not. I personally believe it is possible to create an animal-like or human-like intelligence, without consciousness existing in the system. I personally would still describe whatever processing that system is doing as 'thinking'. Others believe in "substrate independence"; they think any such system must be consciousness.
(Sneaking a bit of belief in here, to me "substrate independence" is a more extreme position than the idea that a system could be made which is intelligent but not conscious, hence I find it implausible.)
> the team wants to tackle a major and common misconception: that students think that ANN systems learn, recognise, see, and understand, when really it’s all just maths
This is completely idiotic. Do these people actually believe that showing it can't be actual thought because it is described by math?
By every scientific measure we have the answer is no. It’s just electrical current taking the path of least resistance through connected neurons mixed with cell death.
The fact a human brain peaks at IQ around 200 is fascinating. Can the scale even go higher? It would seem no since nothing has achieved a higher score it must not exist.
The IQ scale is constantly adjusted to keep the peak of the curve at 100 and the standard deviation around 15. To say it peaks around 200 is a pretty gross misunderstanding of what IQ means.
I think there's a huge divide between the type of people on HN and everyone else, in whether you might know this intuitively or not. Intuition is based on the sum of your experiences.
It's unfortunate that there's so little (none in the article, just 1 comment here as of this writing) mention of the Turing Test. The whole premise of the paper that introduced that was that "do machines think" is such a hard question to define that you have to frame the question differently. And it's ironic that we seem to talk about the Turing Test less than ever now that systems almost everyone can access can arguably pass it now.
> “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.” ~ Edsger W. Dijkstra
The point of the Turing Test is that if there is no extrinsic difference between a human and a machine the intrinsic difference is moot for practical purposes. That is not an argument to whether a machine (with linear algebra, machine learning, large language models, or any other method) can think or what constitutes thinking or consciousness.
The Chinese Room thought experiment is a compliment on the intrinsic side of the comparison: https://en.wikipedia.org/wiki/Chinese_room
The Chinese Room is a pretty useless thought exercise I think. It's an example which if you believe machines can't think seems like an utterly obvious result, and if you believe machines can think it's just obviously wrong.
>And it's ironic that we seem to talk about the Turing Test less than ever now that systems almost everyone can access can arguably pass it now.
Has everyone hastily agreed that it has been passed? Do people argue that a human can't figure out it's talking to an LLM if the user is aware that LLMs exist in the world and is aware of their limitations and that the chat log is able to extend to infinity ( "infinity" is a proxy here for any sufficient time, it could be minutes, days, months, or years)?
In fact, it is blindly easy for these systems to fail the Turing test at the moment. No human would have the patience to continue a conversation indefinitely without telling the person on the other side to kindly fuck off.
Provocative title with a much more reasoned lede.
I'm pretty sure a set of workshops isn't ACTUALLY going to solve a problem that philosophers have been at each other's throats for for the past half century.
But BOY does it get people talking!
Both sides of the debate have capital-O Opinions, and how else did you want to drum up interest for a set of mathematics workshops. O:-)
Yes, that word 'think' is a hell of a distraction. We've replaced that bit of the title with a less provocative / more interesting phrase from the article.
Hopefully we can talk about the actual math and stuff (although the article doesn't go into much of that).
artificial intelligence in place of artificial knowledge is confusing distraction too, at least artificial is a good reminder since beginning
(IMHO its not provocative but well catching a point.. about so called "intelligence" - what if we could look for intelligent knowledge - made in not statistic but semantic(?) and converging way - instead - of being distracted, afraid and.. outpriced, peanuts for the essence it doesn't have ?)
I wish I would've learned about ANNs in elementary school. It looks like a worthwhile and cool lesson package, if only they'd do away with the idiotic dogma...
I took an AI class in 2010 and even then was mostly taught that neural networks aren't very useful and you should use random forests and MCMC.
Don't think this is very good - more of a report of their activities. Underdelivers on the headline.
[stub for offtopicness]
(in this case, thinkiness)
I love the idea of educating students on the math behind AI to demystify them. But I think it's a little weird to assert "AI is not magic and AI systems do not think. It’s just maths." Equivalent statements could be made about how human brains are not magic, just biology - yet I think we still think.
I agree saying "they don't think" and leaving it at that isn't particularly useful or insightful, it's like saying "submarines don't swim" and refusing to elaborate further. It can be useful if you extend it to "they don't think like you do". Concepts like finite context windows, or the fact that the model is "frozen" and stateless, or the idea that you can transfer conversations between models are trivial if you know a bit about how LLMs work, but extremely baffling otherwise.
> Concepts like
> finite context windows
like a human has
> or the fact that the model is "frozen" and stateless,
much like a human adult. Models get updated at a slower frequency than humans. AI systems have access to fetch new information and store it for context.
> or the idea that you can transfer conversations between models are trivial
because computers are better-organized than humanity.
> much like a human adult.
it doesn't sound like you really understand what these statements mean. if LLMs are like any humans it's those with late stage dementia, not healthy adults
> much like a human adult.
I do hope you're able to remember what you had for lunch without incessantly repeating it to keep it in your context window
My context window is about a day. I can remember what I had for lunch today, and sometimes what I had for lunch yesterday. Beyond that, my lunches are gone from my context window and are only in my training data. I have vague ideas about what dishes I ate, but don't remember what days specifically. If I had to tell you what separate dishes I ate in the same meal, I don't have specific memories of that. I remember I ate fried plantains, and I ate beans & rice. I assume they were on the same day because they are from the same cuisine, and am confident enough that I would bet money on it, but I don't know for certain.
One of my earliest memories is of painting a ceramic mug when I was about 3 years old. The only reason I remember it is because every now and then I think about what my earliest memory is, and then I refresh my memory of it. I used to remember a few other things from when I was slightly older, but no longer do, because I haven't had reasons to think of them.
I don't think humans have specific black and white differences between types of knowledge that way LLMs do, but there is definitely a lot of behavior that is similar to context window vs training data (and a gradient in between). We remember recent things a lot better than less recent things. The quantity of stuff we can remember in our "working memory" is approximately finite. If you try to hold a complex thought in your mind, you can probably do that indefinitely, but if you then try to hold a second equally complex thought as well, you'll often lose the details of the first thought and need to reread or rederive those details.
Wouldn't context be comparable to human short term memory, which could be neurons firing in a certain pattern repeatedly to keep it there?
How would you say human short term memory works if not by repeated firing (similar to repeatedly putting same tokens in over and over)?
A lot of people genuinely can't remember what they did an hour ago, but to be very clear you're implying that an LLM can't "remember" something from an hour, or three hours ago, when it's the opposite.
I can restart a conversation with an LLM 15 days later and the state is exactly as it was.
Can't do that with a human.
The idea that humans have a longer, more stable context window than LLM's, CAN or is even LIKELY to be true given certain activities but please let's be honest about this.
If you talk to someone for an hour about a technical conversation I would guesstimate that 90% of humans would immediately start to lose track of details in about 10 minutes. So they write things down, or they mentally repeat things to themselves they know or have recognized they keep forgetting.
I know this because it's happened continually in tech companies decade after decade.
LLM's have already passed the Turing test. They continue to pass it. They fool and outsmart people day after day.
I'm no fan of the hype AI is receiving, especially around overstating its impact in technical domains, but pretending that LLM's can't or don't consistently perform better than most human adults on a variety of different activities is complete non-sense.
The Turing test was passed in the 80s by Eliza it doesnt mean anything
Why doesn't it mean anything?
I do hope you're able to remember what buttons you just pressed without staring at your hands while doing so to keep it in your working memory
I do hope you're able to remember what was your browser tab 5 tab switches ago without keeping track of it...
It's just provencial nonsense, there's no sound reasoning to it. Reductionism being taken and used as a form of refutation is a pretty common cargo culting behavior I've found.
Overwhelmingly, I just don't think the majority of human beings have the mental toolset to work with ambiguous philosophical contexts. They'll still try though, and what you get out of that is a 4th order baudrillardian simulation of reason.
"Just" is used here as a reductive device. You reduce others to a few sentences.
Sentences constructed of words and representations of ideas defined long before you existed. I question whether you can work with ambiguous contexts as you have had the privilege of them being laid out in language for you already by the time you were born.
From my reference frame you appear to merely be circumlocuting from memory, and become the argument you make about others.
Thinking is undefined so all statements about it are unverifiable.
I would say a different problem:
There's many definitions of "thinking".
AI and brains can do some, AI and brains definitely provably cannot do others, some others are untestable at present, and nobody really knows enough about what human brains do to be able to tell if or when some existing or future AI can do whatever is needed for the stuff we find special about ourselves.
A lot of people use different definitions, and respond to anyone pointing this out by denying the issue and claiming their own definition is the only sensible one and "obviously" everyone else (who isn't a weird pedant) uses it.
This is not a meta-question.
The definition of "thinking" in any of the parent comments or TFA is actually not defined. Like literally no statements are made about what is being tested.
So, if we had that we could actually discuss it. Otherwise it's just opinions about what a person believes thinking is, combined with what LLMs are doing + what the person believes they themselves do + what they believe others do. It's entirely subjective with very low SNR b/c of those confounding factors.
What's a definition of thinking that brains definitely provably can't do?
Halting problem.
There are people who insist that the halting problem "proves" that machines will never be able to think. That this means they don't understand the difference between writing down (or generating a proof of) the halting problem and the implications of the halting problem, does not stop them from using it.
Computing the Kolmorgorov constant?
I don't know that I agree that computation is a variety of thinking. It's certainly influenced by thinking, but I think of thinking as more the thing you do before, after, and in-between the computation, not the actual computation itself.
Statements like "it is bound by the laws of physics" are not "verifiable" by your definition, and yet we safely assume it is true of everything. Everything except the human brain, that is, for which wild speculation that it may be supernatural is seemingly considered rational discussion so long as it satisfies people's needs to believe that they are somehow special in the universe.
True. You need to define "it" before you can verify physics bounds it.
Unicorns are not bound by the laws of physics - because they do not exist.
They are, apparently, proscribed by the totality of the laws of physics. For now.
But every unicorn is bound by the laws of physics.
> it satisfies people's needs to believe that they are somehow special in the universe.
Is it only humans that have this need? That makes the need special, so humans are special in the universe.
It is bound by the same laws of physics as everything else, so no, not special.
I think what many are saying is that of all the things we know best, it's going to be the machines we build and their underlying principles.
We don't fully understand how brains work, but we know brains don't function like a computer. Why would a computer be assumed to function like a brain in any way, even in part, without evidence and just hopes based on marketing? And I don't just mean consumer marketing, but marketing within academia as well. For example, names like "neural networks" have always been considered metaphorical at best.
What has it got to do with anything whether brains function like computers? This is only relevant if you define thinking as something only the brain can do, and then nothing that doesn't work like a brain can think. This would be like defining flight as "what birds do" and then saying airplanes can't fly because they don't work like birds.
And then what do you even mean by "a computer?" This falls into the same trap because it sounds like your statement that brains don't function like a computer is really saying "brains don't function like the computers I am familiar with." But this would be like saying quantum computers aren't computers because they don't work like classical computers.
To use your own example, it's relevant because the definition of "flight" that we apply to planes is not as versatile as the one we apply to birds.
To put this in terms of "results", because that's what your way of thinking insists upon, a plane does not take off and land the way a bird does. This limits a plane's practicality to such an extent that a plane is useless for transportation without all the infrastructure you're probably ignoring with your argument. You might also be ignoring all the side effects planes bring with them.
Would you not agree that if we only ever wanted "flight" for a specific use case that apparently only birds can do after evaluating what a plane cannot do, then planes are not capable of "flight"?
This is the very same problem with "thought" in terms of AI. We're finding it's inadequate for what we want the machine to do. Not only is it inadequate for our current use cases, and not only is it inadequate now, but it will continue to be inadequate until we further pin down what "thought" is and determine what lies beyond the Church-Turing thesis.
https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis#P...
Relevant quote: "B. Jack Copeland states that it is an open empirical question whether there are actual deterministic physical processes that, in the long run, elude simulation by a Turing machine; furthermore, he states that it is an open empirical question whether any such processes are involved in the working of the human brain"
Do you think that thinking is undefinable ? If thinking is definable, then all statements about it aren't unverifiable.
Caveat: if thinking is definable, then not all statements about it are unverifiable.
Yes, that's a problem of me not being a native english speaker. "All x aren't y" may mean "not all x are y" in my tongue. Not a single x is y is more what we would say in the previous case. But in our case we would say there are x that aren't y.
If thinking is definable, it is wrong that all statements about it are unverifiable (i.e. there are statements about it that are verifiable.)
Well, basic shit.
Is this some self refuting sentence?
I think they meant "Cannot evaluate : (is <undefined> like x ?), argument missing"
edit : Thinking is undefined, statements about undefined cannot be verified.
is a meta-level grammar the same as an object-level grammar?
Is reasoning undefined? That's what usually meant by "thinking".
Formal reasoning is defined, informal reasoning very much isn't.
At the end of the day most people would agree that if something is able to solve a problem without a lookup table / memorisation that it used reasoning to reach the answer. You are really just splitting hairs here.
What do "most" people thinking about LLMs, then?
The "hair-splitting" underlies the whole GenAI debate.
The difference between thinking and reasoning is that I can "think" that Elvis is still alive, Jewish space lasers are responsible for California wildfires, and Trump was re-elected president in 2020, but I cannot "reason" myself into those positions.
It ties into another aspect of these perennial threads, where it is somehow OK for humans to engage in deluded or hallucinatory thought, but when an AI model does it, it proves they don't "think."
>Equivalent statements could be made about how human brains are not magic, just biology - yet I think we still think.
They're not equivalent at all because the AI is by no means biological. "It's just maths" could maybe be applied to humans but this is backed entirely by supposition and would ultimately just be an assumption of its own conclusion - that human brains work on the same underlying principles as AI because it is assumed that they're based on the same underlying principles as AI.
Unless you're supposing something mystical or supernatural about how brains work, then yes, it is "just" math, there is nothing else it could be. All of the evidence we have shows it's an electrochemical network of neurons processing information. There's no evidence that suggests anything different, or even the need for anything different. There's no missing piece or deep mystery to it.
It's on those who want alternative explanations to demonstrate even the slightest need for them exists - there is no scientific evidence that exists which suggests the operation of brains as computers, as information processors, as substrate independent equivalents to Turing machines, are insufficient to any of the cognitive phenomena known across the entire domain of human knowledge.
We are brains in bone vats, connected to a wonderful and sophisticated sensorimotor platform, and our brains create the reality we experience by processing sensor data and constructing a simulation which we perceive as subjective experience.
The explanation we have is sufficient to the phenomenon. There's no need or benefit for searching for unnecessarily complicated alternative interpretations.
If you aren't satisfied with the explanation, it doesn't really matter - to quote one of Neil DeGrasse Tyson's best turns of phrase: "the universe is under no obligation to make sense to you"
If you can find evidence, any evidence whatsoever, and that evidence withstands scientific scrutiny, and it demands more than the explanation we currently have, then by all means, chase it down and find out more about how cognition works and expand our understanding of the universe. It simply doesn't look like we need anything more, in principle, to fully explain the nature of biological intelligence, and consciousness, and how brains work.
Mind as interdimensional radios, mystical souls and spirits, quantum tubules, none of that stuff has any basis in a ruthlessly rational and scientific review of the science of cognition.
That doesn't preclude souls and supernatural appearing phenomena or all manner of "other" things happening. There's simply no need to tie it in with cognition - neurotransmitters, biological networks, electrical activity, that's all you need.
>it doesn't really matter - to quote one of Neil DeGrasse Tyson's best turns of phrase: "the universe is under no obligation to make sense to you"
Right back at you, brochacho. I'm not the one making a positive claim here. You're the one who insists that it must work in a specific way because you can't conceive of any alternatives. I have never seen ANY evidence or study linking any existent AI or computer system to human cognition.
>There's no need or benefit for searching for unnecessarily complicated alternative interpretations.
Thanks, if it's alright with you I might borrow this argument next time somebody tries to tell me the world isn't flat.
>It simply doesn't look
That's one of those phrases you use when you're REALLY confident that you know what you're talking about.
> like we need anything more, in principle, to fully explain the nature of biological intelligence, and consciousness, and how brains work.
Please fully explain the nature of intelligence, consciousness, and how brains work.
>Mind as interdimensional radios, mystical souls and spirits, quantum tubules, none of that stuff has any basis in a ruthlessly rational and scientific review of the science of cognition.
well i definitely never said anything even remotely similar to that. If i didn't know any better i might call this argument a "hallucination".
AI operates alot like trees do as they are both using maths under the hood.
This is the point, we don't know the delta between brains and AI any assumption is equivalent to my statement.
Math is a superset of both processes (can model/implement both), but that doesn't imply that they are equivalent.
Well, a better retort would be "Human brains are not magic, just physics. Protons, neutrons and electrons don't think".
But I think most people get what GP means.
Until you can define what thinking is, you can't assert that particles don't think (panpsychism).
Panpsychism is actually quite reasonable in part because it changes the questions you ask. Instead of “does it think” you need to ask “in what ways can it think, and in what ways is it constrained? What types of ‘experience/qualia’ can this system have, and what can’t it have?”
When you think in these terms, it becomes clear that LLMs can’t have certain types of experiences (eg see in color) but could have others.
A “weak” panpsychism approach would just stop at ruling out experience or qualia based on physical limitations. Yet I prefer the “strong” pansychist theory that whatever is not forbidden is required, which begins to get really interesting (would imply that for example an LLM actually experiences the interaction you have with it, in some way).
But parent didn't try to apply "it's just maths" to humans. He said one could just as easily say, as some do: "Humans are just biology, hence they're not magic". Our understanding of mathematics, including the maths of transformer models is limited, just as our understanding of biology. Some behaviors of these models have taken researches by surprise, and future surprises are not at all excluded. We don't know exactly how far they will evolve.
As for applying the word thinking to AI systems, it's already in common usage and this won't change. We don't have any other candidate words, and this one is the closest existing word for referencing a computational process which, one must admit, is in many ways (but definitely not in all ways) analogous to human thought.
Human brains might not be explained by the same type of math AI is explained with, but it will be some kind of math...
There's no reason to believe this to be the case. Godel says otherwise.
Human brains and experiences seem to be constrained by the laws of quantum physics, which can be simulated to arbitrary fidelity on a computer. Nit sure where Godel’s incompleteness theory would even come in here…
how are we going to deduce/measure/know the initialization and rules for consciousness? do you see any systems as not encodable/simulatable by quantum?
I think you are asking whether consciousness might be a fundamentally different “thing” from physics and thus hard or impossible to simulate.
I think there is abundant evidence that the answer is ‘no’. The main reason is that consciousness doesn’t give you new physics, it follows the same rules and restrictions. It seems to be “part of” the standard natural universe, not something distinct.
Brain damage? If thought was outside physics, it would be a bit more durable than Humpty Dumpty.
Please explain, because this interpetation of "Godel" is highly nonstandard.
you may consider reading I am a strange loop for that, which can do far better justification than myself
if there's surely no algo to solve the halting problem, why would there be maths that describes consciousness?
Can you look at any arbitrary program and tell if it halts without running it indefinitely? If so, you should explain how and collect your Nobel. Telling everybody whether the Collatz conjecture is correct is a good warm up. If not, you can’t solve the halting program either. What does that have to do with consciousness though?
Having read “I Am a Strange Loop” I do not believe Hofstadter indicates that the existence of Gödel’s theorem precludes consciousness being realizable on a Turing machine. Rather if I recall correctly he points out that as a possible argument and then attempts to refute it.
On the other hand Penrose is a prominent believer that human’s ability to understand Gödel’s theorem indicates consciousness can’t be realized on a Turing machine but there’s far from universal agreement on that point.
per halting problem: any system capable of self reference has unprovable (un)truths, the system can not be complete and consistent. consciousness falls under this umbrella
I'll try and ask OG q more clearly: why would the brain, consciousness, be formalizable?
I think there's a yearn view nature as adhering to an underlying model, and a contrary view that consciousness is transcendental, and I lean towards the latter
> that human brains work on the same underlying principles as AI
That wasn't the assumption though, it was only that human brains work by some "non-magical" electro-chemical process which could be described as a mechanism, whether that mechanism followed the same principles of AI or not.
Straw man. The person who you're responding to talked about "equivalent statements" (emphasis added), whereas you appear to be talking about equivalent objects (AIs vs. brains), and pointing out the obvious flaw in this argument, that AIs aren't biology. The obvious flaw in the wrong argument, that is.
Yeah. This whole AI situation has really exposed how bad most people are at considering the ontological and semantic content of the words they use.
Indeed, people confidently assert as established fact things like "brains are bound by the laws of physics" and therefore "there can't be anything special" about them, so "consciousness is an illusion" and "the mind is a computer", all with absolute conviction but with very little understanding of what physics and maths really do and do not say about the universe. It's a quasi-religious faith in a thing not fully comprehended. I hope in the long run some humility in the face of reality will eventually be (re)learned.
If your position is that brains are not actually bound by the laws of physics -- that they operate on some other plane of existence unbound by any scientifically tested principle -- then it is not only your ideological opposites who have quasi-religious faith in a thing not fully comprehended.
My "position" isn't remotely that. The problem with "brains are bound by the laws of physics" isn't that there's something special about brains. It's that physics doesn't consist of "laws" that things are "bound" by. It consists of theories that attempt to describe.
These theories are enormously successful, but they are also known to be variously incomplete, inconsistent, non-deterministic, philosophically problematic, open to multiple interpretations and only partially understood in their implications, with links between descriptions of things at different scales a particularly challenging and little understood topic. The more you learn about physics (and while I'm no physicist, I have a degree in the subject and have learned a great deal more since) the more you understand the limits of what we know.
Anybody who thinks there's no mystery to physics just doesn't know much about it. Anybody who confidently asserts as fact things like "the brain consists of protons, neutrons and electrons so it's impossible for it to do anything a computer can't do" is deducing things from their own ignorance.
This. People do not understand the implications of the most basic facts of modern science. Gravitation is instantaneous action at a distance via an "occult" force (to quote Newton's contemporaries).
Lot's of assumptions about humanity and how unique we are constantly get paraded in this conversation. Ironically, the people who tout those perspectives are the least likely to understand why we're really not all that "special" from a very factual and academic perspective.
You'd think it would unlock certain concepts for this class of people, but ironically, they seem unable to digest the information and update their context.
A large number of adults I encounter are functionally illiterate, including business people in very high up positions. They are also almost 100% MATHEMATICALLY illiterate, not only unable to solve basic algebra and geometry problems, but completely unable to reason about statistical and probabilistic situations encountered in every day life. This is why gambling is so popular and why people are constantly fooled by politicians. It's bad enough to be without words in the modern world, but being without numbers makes you vulnerable to all manner of manipulations.
Gambling exists more because of people dopamine systems than math...though I get the overall drift. People are fooled by politicians because ?? Also not really math related I think.
I have yet to hear any plausible definition of "thought" that convincingly places LLMs and brains on opposite sides of it without being obviously contrived for that purpose.
A college level approach could look at the line between Math/Science/Physics and Philosophy. One thing from the article that stood out to me was that the introduction to their approach started with a problem about classifying a traffic light. Is it red or green?
But the accompanying XY plot showed samples that overlapped or at least were ambiguous. I immediately lost a lot of my interest in their approach, because traffic lights by design are very clearly red, or green. There aren't mauve or taupe lights that the local populace laughs at and says, "yes, that's mostly red."
I like the idea of studying math by using ML examples. I'm guessing this is a first step and future education will have better examples to learn from.
> traffic lights by design are very clearly red, or green
I suspect you feel this because you are observing the output of a very sophisticated image processing pipeline in your own head. When you are dealing with raw matrixes of rgb values it all becomes a lot more fuzzy. Especially when you encounter different illuminations, exposures and the cropping of the traffic light has noise on it. Not saying it is some intractably hard machine vision problem, because it is not. But there is some variety and fuzzyness there in the raw sensor measurements.
The human mind is not just biology in the same way that LLMs are just math.
Define "think".
We observe through our senses geometric relationships.
Syntax is exactly that; letters, sentences, paragraphs organized in spatial/geometric relationships.
At best thought is recreation of neural networks in the brain which only exist as spatial relationships.
Our senses operate on spatial relationships; enough light to work by, and food relative to stomach to satisfy our biological impetus to survive (which is spatial relationships of biochemistry).
The idea of "thought" as anything but biology makes little sense to me then as a root source is clearly observable. Humanity, roughly, repeats the same social story. All that thought does not seem to be all that useful as we end up in the same place; the majority as serfs of aristocracy.
Personally would prefer less "thought" role-play and more people taking the load of the labor they exploit to enable them to sit and "think".
There's a huge amount of money going to convincing people that AI is magic or better than people. The reprogramming is necessary.
We don't know how brains work.
We really don't know how consciousness works. The popular theories that it's emergent might be proven correct, or might be proven to be like the idea that phlogiston built up in a vacuum, putting out flames.
AI systems compute and humans think. One is math and the other biology.
But they are two different things with overlapping qualities.
It's like MDMA and falling in love. They have many overlapping quantities but no one would claim one is the other.
That's where these threads always end up. Someone asserts, almost violently, that AI does not and/or cannot "think." When asked how to falsify their assertion, perhaps by explaining what exactly is unique about the human brain that cannot and/or will not be possible to emulate, that's the last anyone ever hears from them. At least until the next "AI can't think" story gets posted.
The same arguments that appeared in 2015 inevitably get trotted out, almost verbatim, ten years later. It would be amusing on other sites, but it's just pathetic here.
Consider that you might have become polarized yourself. I often encounter good arguments against current AI systems emulating all essential aspects of human thinking. For example, the fact that they can't learn from few examples, that they can't perform simple mathematical operations without access to external help (via tool calling) or that they have to expend so much more energy to do their magic (and yes, to me they are a bit magical), which makes some wonder if what these models do is a form of refined brute-force search, rather than ideating.
Personally, I'm ok with reusing the word "thinking", but there are dogmatic stances on both sides. For example, lots of people decreeing that biology in the end can't but reduce to maths, since "what else could it be". The truth is we don't actually know if it is possible, for any conceivable computational system, to emulate all essential aspects of human thought. There are good arguments for this (in)possibility, like those presented by Roger Penrose in "the Emperor's new Mind" and "Shadows of the Mind".
For example, the fact that they can't learn from few examples
For one thing, yes, they can, obviously [1] -- when's the last time you checked? -- and for another, there are plenty of humans who seemingly cannot.
The only real difference is that with an LLM, when the context is lost, so is the learning. That will obviously need to be addressed at some point.
that they can't perform simple mathematical operations without access to external help (via tool calling)
But yet you are fine with humans requiring a calculator to perform similar tasks? Many humans are worse at basic arithmetic than an unaided transformer network. And, tellingly, we make the same kinds of errors.
or that they have to expend so much more energy to do their magic (and yes, to me they are a bit magical), which makes some wonder if what these models do is a form of refined brute-force search, rather than ideating.
Well, of course, all they are doing is searching and curve-fitting. To me, the magical thing is that they have shown us, more or less undeniably (Penrose notwithstanding), that that is all we do. Questions that have been asked for thousands of years have now been answered: there's nothing special about the human brain, except for the ability to form, consolidate, consult, and revise long-term memories.
1: E.g., https://arxiv.org/abs/2005.14165 from 2020
> For one thing, yes, they can
That's post-training. The complaint I'm referring to is to the huge amounts of data (end energy) required during training - which is also a form of learning, after all. Sure, there are counter-arguments, for example pointing to the huge amount of non-textual data a child ingests, but these counter-arguments are not waterproof themselves (for example, one can point out that we are discussing text-only tasks). The discussion can go on and on, my point was only that cogent arguments are indeed often presented, which you were denying above.
> there are plenty of humans who seemingly cannot
This particular defense of LLMs has always puzzled me. By this measure, simply because there are sufficiently impaired humans, AGI has already been achieved many decades ago.
> But yet you are fine with humans requiring a calculator to perform similar tasks
I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator.
> To me, the magical thing is that they have shown us, more or less undeniably (Penrose notwithstanding), that that is all we do.
Or, to put it more tersely, they have shown you that that is all we do. Penrose, myself, and lots of others don't see it quite like that. (Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. ;) To me what LLMs do is approximate one aspect of our minds. But I have a strong hunch that the rabbit hole goes much deeper, your assessment notwithstanding.
That's post-training
No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model."
I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator.
So am I. See, for example, Karpathy's discussion of native computation: https://youtu.be/7xTGNNLPyMI?si=Gckcmp2Sby4SlKje&t=6416 (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers.
You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week.
Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly.
Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
> No, it is not.
Yes, it is. You seem to have misunderstood what I wrote. The critique I was pointing to is of the amount of examples and energy needed during model training, which is what the "learning" in "machine learning" actually refers to. The paper uses GPT-3 which had already absorbed all that data and electricity. And the "learning" the paper talks about is arguably not real learning, since none of the acquired skills persists beyond the end of the session.
> So am I.
This is easy to settle. Go check any frontier model and see how far they get with multiplying numbers with tool calling disabled.
> Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
Neither am I resting, nor are there much laurels to rest on, at least compared to someone like Penrose. As for him, give the man a break, he's 94 years old and still sharp as a tack and intellectually productive. You're the one who's resting, imagining you've settled a question which is very much still open. Certainty is certainly intoxicating, so I understand where you're coming from, but claiming anyone who doubts computationalism is not bringing any arguments to the table is patently absurd.
Usually it is the work of the one claiming something to prove it. So if you believe that AI does "think" you are expected to show me that it really does. Claiming it "thinks - prove otherwise" is just bad form and also opens the discussion up for moving the goalposts just as you did with your brain emulation statement. Or you could just not accept any argument made or circumvent it by stating the one trying to disprove your assertion got the definition wrong. There are countless ways to start a bad faith argument using this methodology, hence: Define property -> prove property.
Conversely, if the one asserting something doesn't want to define it there is no useful conversation to be had. (as in: AI doesn't think - I won't tell you what I mean by think)
PS: Asking someone to falsify their own assertion doesn't seem a good strategy here.
PPS: Even if everything about the human brain can be emulated, that does not constitute progress for your argument, since now you'd have to assert that AI emulates the human brain perfectly before it is complete. There is no direct connection between "This AI does not think" to "The human brain can be fully emulated". Also the difference between "does not" and "can not" is big enough here that mangling them together is inappropriate.
So if you believe that AI does "think" you are expected to show me that it really does.
A lot of people seemingly haven't updated their priors after some of the more interesting results published lately, such as the performance of Google's and OpenAI's models at the 2025 Math Olympiad. Would you say that includes yourself?
If so, what do the models still have to do in order to establish that they are capable of all major forms of reasoning, and under what conditions will you accept such proof?
It definietly includes myself, I don't have the interest to stay updated here.
For that matter I have no opinion on if AI does think or not, I simply don't care. Therefore I also really can't answer your question in what more a model has to do to establish that they are thinking (does being able to use all major forms of reasoning constitute the capability of thought to you?). I can say however, that any such proof would have to be on a case-by-case basis given my current understanding on AI is designed.
Then prove to me you are thinking, lest we assume you are a philosophical zombie and need no rights or protections.
Sometimes, because of the consequences of otherwise, the order gets reversed
Well first of all I never claimed that I was capable of thinking (smirk). We also haven't agreed on a definition of "thinking" yet, so as you can read in my previous comment, there is no meaningful conversation to be had. I also don't understand how your oddly aggresive phrasing adds to the conversation, but if it helps you: my rights and protections do not depend on whether I'm able to prove to you that I am thinking. (It also derails the conversation for what it's worth - it's a good strategy in the debating club, but these are about winning or loosing and not about fostering and obtaining knowledge)
Whatever you meant to say with "Sometimes, because of the consequences of otherwise, the order gets reversed" eludes me as well.
If I say I'm innocent, you don't say I have to prove it. Some facts are presumed to be true without burden of evidence because otherwise it could cause great harm.
So we don't require, say, minorities or animals to prove they have souls, we just inherently assume they do and make laws around protecting them.
Thank you for the clarification. If you expect me to justify an action depending on you being innocent, then I actually do need you to prove it. I wouldn't let you sleep in my room assuming you're innocent - or in your words: because of the consequences of otherwise. It feels like you're moving the goalposts here: I don't want to justify an action based on something, i just want to know if something has a specific property.
With regards to the topic: Does AI think? I don't know, but I also don't want to act upon knowing if it does (or doesn't for that matter). In other words, I don't care. The answer could go either way, but I'd rather say that I don't know (especially since "thinking" is not defined). That means that I can assume both and consider the consequences using some heuristic to decide which assumption is better given the action I want to justify doing or not doing. If you want me to believe an AI thinks, you have to prove it, if you want to justify an action you may assume whatever you deem most likely. And if you want to know if an AI thinks, then you literally can't assume it does; simple as that.
Someone asserts, almost religiously, that LLMs do and/or can "think." When asked how to falsify their assertion, perhaps by explaining what exactly is "thinking" in the human brain that can and/or will be possible to emulate...
One mostly sees people aggressively claiming they can’t, ever. On the other side people seem to simply allow that they might, or might eventually.
Err, no, that’s not what’s happening. Nobody, at least in this thread (and most others like it I’ve seen), is confidently claiming LLMs can think.
There are people confidently claiming they can’t and then other people expressing skepticism at their confidence and/or trying to get them to nail down what they mean.
This entire debate over the years is because so many confidently assert that AI can think, or that AI will soon be our God, or our ruler, etc.
Or they just point to the turing test which was the defacto standard test for something so nebulous. And behold: LLM can pass the turing test. So they think. Can you come up with something better (than the turing test)?
But the Turing test (which I concede, LLMs do pass) doesn't test if some system is thinking; it tests if the system can convince an unbiased observer that it is thinking. I cannot come up with a better "is this thing thinking" test, but that doesn't mean that such a test can't exist; I'm sure there are much smarter people then me trying to solve this problem.
When asked how to falsify their assertion, perhaps by explaining what exactly is "thinking" in the human brain that can and/or will be possible to emulate...
... someone else points out that the same models that can't "think" are somehow turning in gold-level performance at international math and programming competitions, making Fields Medalists sit up and take notice, winning art competitions, composing music indistinguishable from human output, and making entire subreddits fail the Turing test.
A couple decades of chess programs nods knowingly: "First time?"
A couple decades of chess programs nods knowingly: "First time?"
Uh huh. Good luck getting Stockfish to do your math homework while Leela works on your next waifu.
LLMs play chess poorly. Chess engines do nothing else at all. That's kind of a big difference, wouldn't you say?
> That's kind of a big difference, wouldn't you say?
To their utility.
Not sure if it matters on the question "thinking?"; even if for the debaters "thinking" requires consciousness/qualia (and that varies), there's nothing more than guesses as to where that emerges from.
Terr_ was agreeing with you and highlighting how old the debate is.
Highlighting, yes, agreeing, no.
For my original earlier reply, the main subtext would be: "Your complaint is ridiculously biased."
For the later reply about chess, perhaps: "You're asserting that tricking, amazing, or beating a human is a reliable sign of human-like intelligence. We already know that is untrue from decades of past experience."
You're asserting that tricking, amazing, or beating a human is a reliable sign of human-like intelligence.
I don't know who's asserting that (other than Alan Turing, I guess); certainly not me. Humans are, if anything, easier to fool than our current crude AI models are. Heck, ELIZA was enough to fool non-specialist humans.
In any case, nobody was "tricked" at the IMO. What happened there required legitimate reasoning abilities. The burden of proof falls decisively on those who assert otherwise.
god of the gaps
Exactly. As soon as a model does something it "wasn't supposed to be able to do," two gaps open up on either side.
Computers can perform math and numerous other tasks billions of times faster than humans, whats your point?
This is exactly the problem. Claims about AI are unfalsifiable, thus your various non-sequiturs about AI 'thinking'.
I feel like these conversations really miss the mark: whether an LLM thinks or not is not a relevant question. It is a bit like asking “what color is an Xray?” or “what does the number 7 taste like?”
The reason I say this is because an LLM is not a complete self-contained thing if you want to compare it to a human being. It is a building block. Your brain thinks. Your prefrontal cortex however is not a complete system and if you somehow managed to extract it and wire it up to a serial terminal I suspect you’d be pretty disappointed in what it would be capable of on its own.
I want to be clear that I am not making an argument that once we hook up sensory inputs and motion outputs as well as motivations, fears, anxieties, desires, pain and pleasure centers, memory systems, sense of time, balance, fatigue, etc. to an LLM that we would get a thinking feeling conscious being. I suspect it would take something more sophisticated than an LLM. But my point is that even if an LLM was that building block, I don’t think the question of whether it is capable of thought is the right question.
> The reason I say this is because an LLM is not a complete self-contained thing if you want to compare it to a human being.
The AI companies themselves are the ones drawing the parallels to a human being. Look at how any of these LLM products are marketed and described.
Is it not within our capacity on HN to ignore whatever the marketers say and speak to the underlying technology?
There was around 500 comments on the OpenAI + Disney story, so the evidence points to "no".
Given the context of the article, why would you ignore that? It's important to discuss the underlying technology in the context in which it's being sold to the public at large.
they are discussing it?
Why do people call is "Artificial Intelligence" when it could be called "Statistical Model for Choosing Data"?
"Intelligence" implies "thinking" for most people, just as "Learning" in machine learning implies "understanding" for most people. The algorithms created neither 'think' nor 'understand' and until you understand that, it may be difficult to accurately judge the value of the results produced by these systems.
If we say “artificial flavoring”, we have a sense that it is an emulation of something real, and often a poor one.
Why, when we use the term for AI, do we skip over this distinction and expect it to be as good as the original—- or better?
That wouldn’t be artificial intelligence, it would just be the original artifact: “intelligence”.
Actually I think the name is apt. It's artificial. It's like how an "artificial banana" isn't actually a banana. It doesn't have to be real thinking or real learning, it just has to look intelligent (which it does).
The term was coined in 1955 to describe "learning or any other feature of intelligence" simulated by a machine [1]. The same proposal does list using natural language as one of the aspects of "the artificial intelligence problem"
It's not a perfect term, but we have been using it for seven full decades to include all of machine learning and plenty of things even less intelligent
1: https://www-formal.stanford.edu/jmc/history/dartmouth/dartmo...
How do you feel about Business Intelligence as a term?
Same way I feel about 'Military Intelligence' :-). Both of those phrases use the 'information gathering and analysis' definition of intelligence rather than the 'thinking' definition.
Like the old saying:
Military justice is to justice as military music is to music
You should take your complaints to OpenAI, who constantly write like LLMs think in the exact same sense as humans; here a random example:
> Large language models (LLMs) can be dishonest when reporting on their actions and beliefs -- for example, they may overstate their confidence in factual claims or cover up evidence of covert actions
They have a product to sell based on the idea AGI is right around the corner. You can’t trust Sam Altman as far as you can throw him.
Still, the sales pitch has worked to unlock huge liquidity for him so there’s that.
Still making predictions is a big part of what brains do though not the only thing. Someone wise said that LLM intelligence is a new kind of intelligence, like how animal intelligence is different from ours but is still intelligence but needs to be characterized to understand differences.
> Someone wise said that LLM intelligence is a new kind of intelligence
So long as you accept the slide ruler as a "new kind of intelligence" everything will probably work out fine, it's the Altmannian insistence that only the LLM is of the new kind that is silly.
This is the underlying problem behind syncophantcy.
I saw a YouTube video about a investigative youtuber Eddy Burback who very easily convinced chat4 that he should cut off all contact with friends and family, move to a cabin in the desert, eat baby food, wrap himself in alfoil, etc just feeding his own (faked) mistakes and delusions. "What you are doing is important, trust your instincts".
Wven if AI could hypothetically be 100x as smart as a human under the hood, it still doesn't care. It doesn't do what it thinks it should, it doesn't do what it needs to do, it does what we train it to.
We train in humanities weaknesses and follies. AI can hypothetically exceed humanity in some respects, but in other respects it is a very hard to control power tool.
AI is optimised, and optimised functions always "hack" the evaluation function. In the case of AI, the evaluation function includes human flaws. AI is trained to tell us what we want to hear.
Elon Musk sees the problem, but his solution is to try to make it think more like him, and even if that succeeds it just magnifies his own weaknesses.
Has anyone read the book criticising Ray Dalio? He is a very successful hedge fund manager, who decided that he could solve the problem of finding a replacement by psychology evaluation and training people to think like him. But even his smartest employees didn't think like him, they just (reading between the lines) gamed his system. Their incentives weren't his incentives - he could demand radical honesty and integrity but that doesn't work so well when he would (of course) reward the people who agreed with him, rather than the people who would tell him he was screwing up. His organisation (apparently) became a bunch of even more radical syncopants due to his efforts to weed out syncophantcy.
The part that's most infuriating is that we don't have to speculate at all. Any discussion or philosophizing beyond the literal computer science is simply misinformation.
There's absolutely no similarity between what computer hardware does and what a brain does. People will twist and stretch things and tickle the imagination of the naive layperson and that's just wrong. We seriously have to cut this out already.
Anthropomorphizing is dangerous even for other topics, and long understood to be before computers came around. Why do we allow this?
The way we talk about computer science today sounds about as ridiculous as invoking magic or deities to explain what we now consider high school physics or chemistry. I am aware that the future usually sees the past as primitive, but why can't we try to seem less dumb at least this time around?
> There's absolutely no similarity between what computer hardware does and what a brain does.
But at very least there's also no similarity between what computer hardware does and what even the simplest of LLMs do. They don't run on eg. x86_64 , else qemu would be sufficient for inferencing.
Similarity of the hardware is absolutely irrelevant when we're talking about emergent behavior like "thought".
You can replicate all calculations done by LLMs with pen and paper. It would take ages to calculate anything, but it's possible. I don't think that pen and paper will ever "think", regardless of how complex the calculations involved are.
The official name is https://en.wikipedia.org/wiki/Chinese_room
The opinions are exactly the same than about LLM.
And the counter argument is also exactly the same. Imagine you take one neuron from a brain and replace it with an artificial piece of electronics (e.g. some transistors) that only generates specific outputs based on inputs, exactly like the neuron does. Now replace another neuron. And another. Eventually, you will have the entire brain replaced with a huge set of fundamentally super simple transistors. I.e. a computer. If you believe that consciousness or the ability to think disappears somewhere during this process, then you are essentially believing in some religious meta-physics or soul-like component in our brains that can not be measured. But if it can not be measured, it fundamentally can not affect you in any way. So it doesn't matter for the experiment in the end, because the outcome would be exactly the same. The only reason you might think that you are conscious and the computer is not is because you believe so. But to an outsider observer, belief is all it is. Basically religion.
It seems like the brain "just" being a giant number of neurons is an assumption. As I understand it's still an area of active research, for example the role of glial cells. The complete function may or may not be pen and paper-able.
There are indeed many people trying to justify this magical thinking by seeking something, anything in the brain that is out of the ordinary. They've been unsuccessful so far.
Penrose comes to mind, he will die on the hill that the brain involves quantum computations somehow, to explain his dualist position of "the soul being the entity responsible for deciding how the quantum states within the brain collapse, hence somehow controlling the body" (I am grossly simplifying). But even if that was the case, if the brain did involve quantum computations, those are still, well, computable. They just involve some amount of randomness, but so what? To continue with grandparent's experiment, you'd have to replace biological neurons with tiny quantum computer neurons instead, but the gist is the same.
You wouldn't even need quantum computer neurons. We can simulate quantum nature on normal circuits, albeit not very efficiently. But for the experiment this wouldn't matter. The only important thing would be that you can measure it, which in turn would allow you to replicate it in some non-human circuit. And if you fundamentally can't measure this aspect for some weird reason, you will once again reach the same conclusion as above.
You can simulate it, but you usually use PRNG to decide how your simulated wave function "collapses". So in the spirit of the original thought experiment, I felt it more adequate to replace the quantum part (if it even exists) by another actually quantum part. But indeed, using fake quantum shouldn't change a thing.
> The complete function may or may not be pen and paper-able.
Would you mind expanding on this? At a base read, it seems you implying magic exists.
It could well be the case that the brain can be simulated, but presently we don't know exactly what variables/components must be simulated. Does ongoing neuroplasticity for example need to be a component of simulation? Is there some as of yet unknown causal mechanisms or interactions that may be essential?
None of those examples cannot be done on pen and paper or otherwise simulated with a different medium, though.
AFAICT, your comment above would need some mechanism that is physically impossible and incalculable to make the argument, and then somehow have that happen in a human brain despite being physically impossible and incalculable.
> component in our brains that can not be measured.
"Can not be measured", probably not. "We don't know how to measure", almost certainly.
I am capable of belief, and I've seen no evidence that the computer is. It's also possible that I'm the only person that is conscious. It's even possible that you are!
But you are now arguing against a strawman, namely, "it is not possible to construct a computer that thinks".
The argument that was actually made was "LLMs do not think".
A: X, because Y
B: But Y would also imply Z
C: A was never arguing for Z! This is a strawman!
"LLMs cannot think like brains" does not imply "no computer it will ever be possible to construct could think like a brain".
“LLMs cannot think like brains” is “X”.
That appears to be your own assumptions coming into play.
Everything I've seen says "LLMs cannot think like brains" is not dependent on an argument that "no computer can think like a brain", but rather on an understanding of just what LLMs are—and what they are not.
I don’t understand why people say the Chinese Room thing would prove LLMs don’t think, to me it’s obvious that the person doesn’t understand Chinese but the process does, similarly the CPU itself doesn’t understand the concepts an LLM can work with but the LLM itself does, or a neuron doesn’t understand concepts but the entire structure of your brain does
The concept of understanding emerges on a higher level from the way the neurons (biological or virtual) are connected, or the way the instructions being followed by the human in the Chinese room process the information
But really this is a philosophical/definitional thing about what you call “thinking”
Edit: I see my take on this is listed on the page as the “System reply”
If 100 top-notch philosophers disagree with you, that means you get 100 citations from top-notch philosophers. :-P
Check out eg Dennett.... or ... his opionions about Searle; Have fun with eg... this:
"By Searle’s own count, there are over a hundred published attacks on it. He can count them, but I guess he can’t read them, for in all those years he has never to my knowledge responded in detail to the dozens of devastating criticisms they contain;"
https://www.nybooks.com/articles/1995/12/21/the-mystery-of-c...
I don't see the relevance of that argument (which other responders to your post have pointed out as Searle's Chinese Room argument). The pen and paper are of course not doing any thinking, but then the pen isn't doing any writing on its own, either. It's the system of pen + paper + human that's doing the thinking.
The idea of my argument is that I notice that people project some "ethereal" properties over computations that happen in the... computer. Probably because electricity is involved, making things show up as "magic" from our point of view, making it easier to project consciousness or thinking onto the device. The cloud makes that even more abstract. But if you are aware that the transistors are just a medium that replicates what we already did for ages with knots, fingers, and paint, it gets easier to see them as plain objects. Even the resulting artifacts that the machine produces are only something meaningful from our point of view, because you need prior knowledge to read the output signals. So yeah, those devices end up being an extension of ourselves.
Your view is missing the forest for the trees. You see individual objects but miss the aggregate whole. You have a hard time conceiving of how exotic computers can be conscious because we are scale chauvinists by design. Our minds engage with the world on certain time and length scales, and so we naturally conceptualize our world based on entities that exist on those scales. But computing is necessarily scale independent. It doesn't matter to the computation if it is running on some 100GHz substrate or .0001Hz. It doesn't matter if its running on a CPU chip the size of a quarter or spread out over the entire planet. Computation is about how information is transformed in semantically meaningful ways. Scale just doesn't matter.
If you were a mind supervening on the behavior of some massive time/space scale computer, how would you know? How could you tell the difference between running on a human making marks with pen and paper and running on a modern CPU? Your experience updates based on information transformations, not based on how fast the fundamental substrate is changing. When your conscious experience changes, that means your current state is substantially different from your prior state and you can recognize this difference. Our human-scale chauvinism gets in the way of properly imagining this. A mind running on a CPU or a large collection of human computers is equally plausible.
A common question people like to ask is "where is the consciousness" in such a system. This is an important question if only because it highlights the futility of such questions. Where is Microsoft Word when it is running on my computer? How can you draw a boundary around a computation when there are a multitude of essential and non-essential parts of the system that work together to construct the relevant causal dynamic. It's just not a well-defined question. There is no one place where Microsoft Word occurs nor is there any one place where consciousness occurs in a system. Is state being properly recorded and correctly leveraged to compute the next state? The consciousness is in this process.
If you put a droplet of water in a warm bowl every 12 hours, the bowl will remain empty as the water will evaporate. That does not mean that if you put a trillion droplets in every twelve hours it will still remain empty.
It will also not be empty if I put the bowl in the sea, which is a remark about the nature of thoughthat that proves exactly as much as your comment.
The point I was trying to make was that the time you use to perform the calculation may change whether there is an "experience" on behalf of the calculation. Without specifying the basis if subjectivity, you can't rule anything out as far as what matters and what doesn't. Maybe the speed or locality with which the calculations happen matters. Like the water drops: given the same amount of time, eventually all the water will evaporate in either case leading to the same end state, but the the intermediate states are very different.
https://xkcd.com/505/
You can replicate the entire universe with pen and paper (or a bunch of rocks). It would take an unimaginably long time, and we haven't discovered all the calculations you'd need to do yet, but presumably they exist and this could be done.
Does that actually make a universe? I don't know!
The comic is meant to be a joke, I think, but I find myself thinking about it all the time!!!
Even worse, as we are part of the universe, we would need to simulate ourselves and the very simulation that we are creating. You would also need to replicate the simulation of the simulation, leading to an eternal loop that would demand infinite matter and time (and would still not be enough!). Probably, you can't simulate something while being part of it.
It doesn’t need to be our universe, just a universe.
The question is, are the people in the simulated universe real people? Do they think and feel like we do—are they conscious? Either answer seems like it can’t possibly be right!
You can simulate a human brain on pen and paper too.
> You can simulate a human brain on pen and paper too.
That's an assumption, though. A plausible assumption, but still an assumption.
We know you can execute an LLM on pen and paper, because people built them and they're understood well enough that we could list the calculations you'd need to do. We don't know enough about the human brain to create a similar list, so I don't think you can reasonably make a stronger statement than "you could probably simulate..." without getting ahead of yourself.
I can make a claim much stronger than "you could probably" The counterclaim here is that the brain may not obey physical laws that can be described by mathematics. This is a "5G causes covid" level claim. The overwhelming burden of proof is on you.
There are some quantum effects in the brain (for some people, that's a possible source of consciousness). We can simulate quantum effects, but here comes the tricky part: even if our simulation matches the probability, say 70/30 of something happening, what guarantees that our simulation would take the same path as the object being simulated?
We don't have to match the quantum state since the brain still produces an valid output regardless of what each random quantum probability ended up as. And we can include random entropy in a LLM too.
This is just non-determinism. Not only can't your simulation reproduce the exact output, but neither can your brain reproduce its own previous state. This doesn't mean it's a fundamentally different system.
Consider for example Orch OR theory. If it or something like it were to be accurate, the brain would not "obey physical laws that can be described by mathematics".
>Consider for example Orch OR theory
Yes, or what about leprechauns?
Orch OR is probably wrong, but the broader point is that we still don’t know which physical processes are necessary for cognition. Until we do, claims of definitive brain simulability are premature.
the transition probability matrices don't obey the laws of statistics?
This is basically the Church-Turing thesis and one of the motivations of using tape(paper) and an arbitrary alphabet in the Turing machine model.
It's been kinda discussed to oblivion in the last century, interesting that it seems people don't realize the "existing literature" and repeat the same arguments (not saying anyone is wrong).
The simulation isn't an operating brain. It's a description of one. What it "means" is imposed by us, what it actually is, is a shitload of graphite marks on paper or relays flipping around or rocks on sand or (pick your medium).
An arbitrarily-perfect simulation of a burning candle will never, ever melt wax.
An LLM is always a description. An LLM operating on a computer is identical to a description of it operating on paper (if much faster).
What makes the simulation we live in special compared to the simulation of a burning candle that you or I might be running?
That simulated candle is perfectly melting wax in its own simulation. Duh, it won't melt any in ours, because our arbitrary notions of "real" wax are disconnected between the two simulatons.
They do have a valid subtle point though.
If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
Being a functionalist ( https://en.wikipedia.org/wiki/Functionalism_(philosophy_of_m... ) myself, I don't know the answer on the top of my head.
>If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
I can smell a "real" candle, a "real" candle can burn my hand. The term real here is just picking out a conceptual schema where its objects can feature as relata of the same laws, like a causal compatibility class defined by a shared causal scope. But this isn't unique to the question of real vs simulated. There are causal scopes all over the place. Subatomic particles are a scope. I, as a particular collection of atoms, am not causally compatible with individual electrons and neutrons. Different conceptual levels have their own causal scopes and their own laws (derivative of more fundamental laws) that determine how these aggregates behave. Real (as distinct from simulated) just identifies causal scopes that are derivative of our privileged scope.
Consciousness is not like the candle because everyone's consciousness is its own unique causal scope. There are psychological laws that determine how we process and respond to information. But each of our minds are causally isolated from one another. We can only know of each other's consciousness by judging behavior. There's nothing privileged about a biological substrate when it comes to determining "real" consciousness.
Right, but doesn't your argument imply that the only "real" consciousness is mine?
I'm not against this conclusion ( https://en.wikipedia.org/wiki/Philosophical_zombie ) but it doesn't seem to be compatible with what most people believe in general.
That's a fair reading but not what I was going for. I'm trying to argue for the irrelevance of causal scope when it comes to determining realness for consciousness. We are right to privilege non-virtual existence when it comes to things whose essential nature is to interact with our physical selves. But since no other consciousness directly physically interacts with ours, it being "real" (as in physically grounded in a compatible causal scope) is not an essential part of its existence.
Determining what is real by judging causal scope is generally successful but it misleads in the case of consciousness.
> If we don't think the candle in a simulated universe is a "real candle", why do we consider the intelligence in a simulated universe possibly "real intelligence"?
A candle in Canada can't melt wax in Mexico, and a real candle can't melt simulated wax. If you want to differentiate two things along one axis, you can't just point out differences that may or may not have any effect on that axis. You have to establish a causal link before the differences have any meaning. To my knowledge, intelligence/consciousness/experience doesn't have a causal link with anything.
We know our brains cause consciousness the way we knew in 1500 that being on a boat for too long causes scurvy. Maybe the boat and the ocean matter, or maybe they don't.
I think the core trouble is that it's rather difficult to simulate anything at all without requiring a human in the loop before it "works". The simulation isn't anything (well, it's something, but it's definitely not what it's simulating) until we impose that meaning on it. (We could, of course, levy a similar accusation at reality, but folks tend to avoid that because it gets uselessly solipsistic in a hurry)
A simulation of a tree growing (say) is a lot more like the idea of love than it is... a real tree growing. Making the simulation more accurate changes that not a bit.
I believe that the important part of a brain is the computation it's carrying out. I would call this computation thinking and say it's responsible for consciousness. I think we agree that this computation would be identical if it were simulated on a computer or paper. If you pushed me on what exactly it means for a computation to physically happen and create consciousness, I would have to move to statements I'd call dubious conjectures rather than beliefs - your points in other threads about relying on interpretation have made me think more carefully about this.
Thanks for stating your views clearly. I have some questions to try and understand them better:
Would you say you're sure that you aren't in a simulation while acknowledging that a simulated version of you would say the same?
What do you think happens to someone whose neurons get replaced by small computers one by one (if you're happy to assume for the sake of argument that such a thing is possible without changing the person's behavior)?
It seems to me that the distinction becomes irrelevant as soon as you connect inputs and outputs to the real world. You wouldn't say that a 737 autopilot can never, ever fly a real jet and yet it behaves exactly the same whether it's up in the sky or hooked up to recorded/simulated signals on a test bench.
Here is a thought experiment:
Build a simulation of creatures that evolve from simple structures (think RNA, DNA).
Now, if in this simulation, after many many iterations, the creatures start talking about consciousness, what does that tell us?
> An arbitrarily-perfect simulation of a burning candle will never, ever melt wax.
It might if the simulation includes humans observing the candle.
So the brain is a mathematical artifact that operates independently from time? It just happens to be implemented using physics? Somehow I doubt it.
The brain follows the laws of physics. The laws of physics can be closely approximated by mathematical models. Thus, the brain can be closely approximated by mathematical models.
It's an open problem whether you can or not.
It’s not that open. We can simulate smaller system of neurons just fine, we can simulate chemistry. There might be something beyond that in our brains for some reason, but it sees doubtful right now
Our brains actually do something, may be the difference. They're a thing happening, not a description of a thing happening.
Whatever that something that it actually does in the real, physical world is produces the cogito in cogito, ergo sum and I doubt you can get it just by describing what all the subatomic particles are doing, any more than a computer or pen-and-paper simulated hurricane can knock your house down, no matter how perfectly simulated.
Doing something merely requires I/O. Brains wouldn't be doing much without that. A sufficiently accurate simulation of a fundamentally computational process is really just the same process.
Why are the electric currents moving in a GPU any less of a "thing happening" than the firing of the neurons in your brain? What you are describing here is a claim that the brain is fundamentally supernatural.
Thinking that making scribbles that we interpret(!!!) as perfectly describing a functioning consciousness and its operation, on a huge stack of paper, would manifest consciousness in any way whatsoever (hell, let's say we make it an automated flip-book, too, so it "does something"), but if you made the scribbles slightly different it wouldn't work(!?!? why, exactly, not ?!?!), is what's fundamentally supernatural. It's straight-up Bronze Age religion kinds of stuff (which fits—the tech elite is full of that kind of shit, like mummification—er, I mean—"cryogenic preservation", millenarian cults er, I mean The Singularity, et c)
Of course a GPU involves things happening. No amount of using it to describe a brain operating gets you an operating brain, though. It's not doing what a brain does. It's describing it.
(I think this is actually all somewhat tangential to whether LLMs "can think" or whatever, though—but the "well of course they might think because if we could perfectly describe an operating brain, that would also be thinking" line of argument often comes up, and I think it's about as wrong-headed as a thing can possibly be, a kind of deep "confusing the map for the territory" error; see also comments floating around this thread offhandedly claiming that the brain "is just physics"—like, what? That's the cart leading the horse! No! Dead wrong!)
Computation doesn't care about its substrate. A simulation of a computation is just a computation.
You're arguing for the existence of a soul, for dualism. Nothing wrong with that, except we have never been able to measure it, and have never had to use it to explain any phenomenon of the brain's working. The brain follows the rules of physics, like any other objects of the material world.
A pen and paper simulation of a brain would also be "a thing happening" as you put it. You have to explain what is the magical ingredient that makes the brain's computations impossible to replicate.
You could connect your brain simulation to an actual body, and you'd be unable to tell the difference with a regular human, unless you crack it open.
> You're arguing for the existence of a soul, for dualism.
I'm not. You might want me to be, but I'm very, very much not.
Parent said replicate, as in deterministically
You're arguing against Functionalism [0], of which I'd encourage you to at least read the Wikipedia page. Why would doing the brain's computations on pen and paper rather than on wetware lead to different outcomes? And how?
Connect your pen and paper operator to a brainless human body, and you got something indistinguishable from a regular alive human.
[0] https://en.wikipedia.org/wiki/Functionalism_%28philosophy_of...
Wouldn't 'thinking' need to be updating the model of reality (LLM is not yet that, just words) - at every step doing again all that extensive calculations as when/to creating/approximating that/better model (learning) ?
Expecting machines to think is.. like magical thinking (but they are good at calculations indeed).
I wish we didn't use the word intelligence in context of LLMs - shortly there is Essence and the rest.. is only slope - into all possible combinations of Markov Chains - may they have sense or not I don't see how part of some calculation could recognize it, or that to be possible from inside (of calculation, that doesn't even consider that).
Aside of artificial knowledge (out of senses, experience, context lengths.. - confabulating but not knowing that), I wish to see an intelligent knowledge - made in kind of semantic way - allowed to expand using not yet obvious (but existing - not random) connections. I wouldn't expect it to think (humans think, digitals calculate). But I would expect it to have a tendency to be coming closer (not further) in reflecting/modeling reality and expanding implications.
Thinking is different than forming long term memories.
An LLM could be thinking in one of two ways. Either between adding each individual token, or collectively across multiple tokens. At the individual token level the physical mechanism doesn’t seem to fit the definition being essentially reflexive action, but across multiple tokens that’s a little more questionable especially as multiple approaches are used.
An LLM ..is calculated ..from language (or from things being said by humans before being true or not). It's not some antropomorfic process what using the word thinking would suggest (to sell well).
> across multiple tokens
- but how many ? how many of them happen in sole person life ? How many in some calculation ? Does it matter, if a calculation doesn't reflect it but stay all the same ? (conversation with.. a radio - would it have any sense ?)
The general public have no issue saying a computer is thinking when you’re sitting there waiting for it to calculate a route or doing a similar process like selecting a chess move.
The connotation is simply an internal process of indeterminate length rather than one of reflexive length. So they don’t apply it when a GPU is slinging out 120 FPS in a first person shooter.
That's right when saying selecting not calculating a chess move - assuming you are outside of Plato's cave (Popper).
But now, I see this: the truth is static and non-profit, but calculating something can be sold again and again, if you have a hammer (processing) everything looks like a nail, to sell well the word thinking had to be used instead of excuse for every time results being different (like the shadows) - then, we can have only things that let someone else keep making profits: JS, LLM, whatever.. (just not.. "XSLT" alike).
(yet, I need to study for your second sentence;)
.. and confront about Prolog or else in recent years - likes: "intended benefit requires an unreasonably (or impossibly?) smart compiler" (https://news.ycombinator.com/item?id=14441045) - isn't quite similar to LLMs, for that, requiring.. impossibly smart users ?? (there were few - assuming they got what they wanted . not peanuts)
- how to prove that humans can argue endlessly like an llm?
- ragebait them by saying AIs don’t think
- …
LLMs don't really think, they emulate their training data. Which has a lot of examples of humans walking through problems to arrive at an answer. So naturally, if we prompt an LLM to do the same, it will emulate those examples (which tend to be more correct).
LLMs are BAD at evaluating earlier thinking errors, precisely because there's not copious examples of text where humans thinking through a problem, screwing up, going back, correcting their earlier statement, and continuing. (a good example catches these and corrects them)
Claude code is actually great at that
Given that the headline is:
> Secondary school maths showing that AI systems don’t think
And the article contains the quotes:
> the team wants to tackle a major and common misconception: that students think that ANN systems learn, recognise, see, and understand, when really it’s all just maths.
> The team is taking very complex ideas and reducing them to such an extent that we can use secondary classroom maths to show that AI is not magic and AI systems do not think.
This is not off topic
If it comes to the correct answer I don't particularly care how it got there.
In most cases, you don’t know if it came to the correct answer.
In every reasonable use case for LLMs verifying the answer is trivial. Does the code do what I wanted it to? Does it link to a source that corroborates the response?
If you're asking for things you can't easily verify you're barking up the wrong tree.
How do you know if it came to the right answer?
It's not always the case, but often verifying an answer is far easier than coming up with the answer in the first place. That's precisely the principle behind the RSA algorithm for cryptography.
Sure, it's easy to check ((sqrt(x-3)+1)/(x/8)) is less than 4. Now do it without calculus.
Very much like this effect https://www.reddit.com/r/opticalillusions/comments/1cedtcp/s... . Shouldn't hide complexity under a truth value.
@dang offtopicness started from using word thinking in place of calculating what is the common objection in this thread.
Yes. But it's offtopic because the presence of a provocative word 'thinking' in the title led to a lot of generic tangents that don't engage with anything interesting in the article, and mostly just express people's pre-existing associations about a controversial point.
Trying to avoid this kind of thing is why the guidelines say things like:
"Eschew flamebait. Avoid generic tangents."
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
https://news.ycombinator.com/newsguidelines.html
A lot of the drama here is due to the ambiguity of what the word 'think' is supposed to mean. One camp associates 'thinking' to consciousness, another does not. I personally believe it is possible to create an animal-like or human-like intelligence, without consciousness existing in the system. I personally would still describe whatever processing that system is doing as 'thinking'. Others believe in "substrate independence"; they think any such system must be consciousness.
(Sneaking a bit of belief in here, to me "substrate independence" is a more extreme position than the idea that a system could be made which is intelligent but not conscious, hence I find it implausible.)
This article doesn’t really show anything near what the title assets.
> the team wants to tackle a major and common misconception: that students think that ANN systems learn, recognise, see, and understand, when really it’s all just maths
This is completely idiotic. Do these people actually believe that showing it can't be actual thought because it is described by math?
Wait until they hear about the physics/maths related to neurons firing!
<think>Ok, the user is claiming that... </think> ....
Do we think?
By every scientific measure we have the answer is no. It’s just electrical current taking the path of least resistance through connected neurons mixed with cell death.
The fact a human brain peaks at IQ around 200 is fascinating. Can the scale even go higher? It would seem no since nothing has achieved a higher score it must not exist.
The IQ scale is constantly adjusted to keep the peak of the curve at 100 and the standard deviation around 15. To say it peaks around 200 is a pretty gross misunderstanding of what IQ means.
3 years ago this is the kind of posts that end up in /r/im14andthisisdeep
[flagged]
Maybe so, but please don't post unsubstantive comments to Hacker News.
I think we all intuitively knew this but it's pretty cool.
I think there's a huge divide between the type of people on HN and everyone else, in whether you might know this intuitively or not. Intuition is based on the sum of your experiences.