> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
Nobody is actually trying to do that, because it's too hard, they don't know enough about how the brain works, and/or they can't compete on efficiency; pick whichever reasons you like, or add more, but don't pretend that AI researchers have figured out how to simulate a brain.
Not really. He said no one who knows anything about the brain thinks it's a binary system.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
You're reading him too narrowly. He's answering the question of whether the brain can be modeled (math or magic), and arguing against it. Or maybe he's arguing for a middle ground. Hard to say exactly, he's not very clear.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
It shows that it doesn't matter. There is nothing a brain can compute that a base 2 system cannot.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
Does it matter if "intelligence and reasoning" are there if what they produce is indistinguishable (or outmatches) what a human could produce?
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
Your last sentence is identical to your first sentence in your other comment. How can we be sure that you are reasoning and not just a stochastic parrot?
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
An LLM is just a brain in a vat.
Our brains would also not reason in such a condition.
Given the right framework that can do miraculous things.
I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
You made me start listening to Critique of Pure Reason [1] :-)
1. https://www.youtube.com/watch?v=cyhXtJN2AQQ
Suppose you put an LLM in a "inference -> observe -> retrain weights" loop.
Supervised training uses this precise loop. The difference between inference and observation is called the loss function.
Models are now starting to show emergent behavior that they were not explicitly trained for. This is leading to the current debate.
Well, yeah, I understand there're all kinds of theories about this. That wasn't my question though.
Sorry, I misinterpreted your question:
Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning?
As:
Do we have evidence humans have an independent set of beliefs not shaped by observation and experience?
Because, that will be a priori or pure reason.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
You cannot have a syllogism without propositions.
Because he's writing about epistemologies: Sets of rules for reasoning over beliefs.
In such systems, a belief is a proposition you accept as true. (provisionally at least)
For example as a scientist, you probably use the epistemology called empiricism all day. In empiricism, beliefs are justified by observation.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.
Yea.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
[1] https://www.researchgate.net/publication/229060046_Telling_m...
[2] https://pubmed.ncbi.nlm.nih.gov/16210542/
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC5986841/
[4] https://pubmed.ncbi.nlm.nih.gov/2016668/
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Emergent behavior in otherwise simple rulesets.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
I think the question is if the "rules" however captured can express reasoning.
My position is that it is not possible.
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
Do you agree that an LLM in a harness is Turing complete?
Seems so to me.
So explain then what you mean by "not possible".
Turing complete does not mean reasoning. Zuse did turing complete computers in 1940s.
Turing complete means being able to compute any computable output.
So what about reasoning is not computable in your mind?
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
The brain is just ones and zeros on salty mush.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
As long as you believe a brain is just a physical system - then it can be simulated to any degree of accuracy with plain ol' binary computers.
Nobody is actually trying to do that, because it's too hard, they don't know enough about how the brain works, and/or they can't compete on efficiency; pick whichever reasons you like, or add more, but don't pretend that AI researchers have figured out how to simulate a brain.
You're arguing about whether it's possible or not. You're saying it's not possible.
It is possible.
Not really. He said no one who knows anything about the brain thinks it's a binary system.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
You're reading him too narrowly. He's answering the question of whether the brain can be modeled (math or magic), and arguing against it. Or maybe he's arguing for a middle ground. Hard to say exactly, he's not very clear.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
No need to straw man, they aren't pretending anything. It's just a fact about computational theory.
Just because I can't count every grain of sand on Earth, doesn't mean sand grains are uncountable. Those are two distinct facts.
The practicality of a binary brain is distinct from the possibility of it.
you can represent floating point numbers in binary
Music isn't binary either, and yet here we are with totally digitized lossless music ecosystem.
What does the ability to encode or decode data in binary have to do with the brain being or not being a binary computer?
It shows that it doesn't matter. There is nothing a brain can compute that a base 2 system cannot.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
Mathematical modelling in binary, is not the same thing as the thing being modelled.
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
Brain is not just ones and zeros, it can be viewed as combination of infinite quantum states.
Infinite? Or "huge number"?
Infinite in the sense that quantum superposition can have infinite range. So it really is a huge number out of infinite possible combinations.
Does it matter if "intelligence and reasoning" are there if what they produce is indistinguishable (or outmatches) what a human could produce?
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
Folks see AI seeing Jesus in burnt toast, and then admit it as one of the folks. :)
How do you know it’s not when looking into the mirror, that we see Jesus in burnt toast?
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Jet planes don't fly by flapping their wings...
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
Birds don't fly by producing thrust out of their asses... what the heck that comparison does even mean?
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
Reading charitably: just because our inventions do something different to nature doesn't mean that our invention is wrong;
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
> there's no global deployed autonomous driving
Love to see the forever moving goalposts. One might say they're autonomously moving..
> Love to see the forever moving goalposts.
Do we have finally reliable autonomous driving? And available outside the bay area or whatever small place compared to the rest of the world?
Human Technology appears to use different principles than nature oftentimes
Don't be fooled into thinking that planes can fly.
They do fly for sure, but they're not birds. This comparison between fly and intelligence is worthy of the most vulgar bar talk
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
Your last sentence is identical to your first sentence in your other comment. How can we be sure that you are reasoning and not just a stochastic parrot?
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
https://archive.ph/ZyCoe
I thought I saw a paper recently explaining that LLMs have a global workspace. Is that not like having the internal state that he's talking about?
There was a blog post from Anthropic saying Claude has a "J space" or something like that.
Why J?
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
An LLM is just a brain in a vat. Our brains would also not reason in such a condition. Given the right framework that can do miraculous things. I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
https://www.finextra.com/blogposting/31255/adaptability-as-e...
Don't be fooled, submarines don't swim.
well, its a matter of definitions.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
forgive typos, i am on mobile.
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
Makes me think how human thinking is mostly recall from past learning/experience versus reasoning.
<unbearable web page to read>
Agreed, especially because it also seems to defeat Reader mode. Try running it through Marky and then previewing: https://heckyesmarkdown.com/preview.cgi?readability=1&inline...
That is amazingly useful tool I didn’t know about until now, thank you.
Sure thing; it’s a Brett Terpstra creation.
Textise is similar and also doubles as a poor man’s text-only browser. https://www.textise.net/referrer
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
The article links to the book „Thinking, Fast and Slow” by Daniel Kahneman which seems interesting. Anyone read it have thoughts?
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
Yep, it's not 'AI', or even just 'I', if anything it is just the 'A' wrapped in buzzwords.
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
You can sort of do this.
For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
Yeah, and red-team/blue-teaming it.
(I'm definitely one of the bigger "AI" skeptics out there but am nonetheless fascinated by these questions).
If AI cannot reason, can humans?
I think the real question is: how do we define reasoning?
My view is that AI can reason, just differently from humans.
But neither do the vast majority of humans.
Garbage In Garbage Out
LLMs lack the holy ghost
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
Yes, this is true—there's no "logic" in the sense of deductive rigor. It's a wonder we animals are capable of it.
I think, "contexting" is apropos.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
It's wild when you think about it, you don't need to reason to solve the hardest math problems that humans failed to solve for decades.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Same for programming, turns out 90% of it is dumb statistical prediction, no reasoning required at all.
Neither do humans!
At least reasoning is not guaranteed.
Would sure be nice though...
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
https://youtu.be/wTSbvYBx4eg?si=MfTBFsE2kA3TE6EE&t=189
We can't even say what human reasoning is. Who could really say what isn't reasoning?
99% of the people treat it as a black box. It gives better answers, you can reverse reason and IDGAF
[flagged]