This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.
This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which translates to fewer total tokens needed to solve problems, and harder problems that a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve many problems (even hard ones, e.g. a flow state).
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
I think this is the core thrust of the joint-embedding predictive architecture that LeCunn has been pursuing.
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
That can make sense though.
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
If you can find this research I would love to see it. That’s extremely interesting.
"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" by Jacob Pfau, William Merrill, and Samuel R. Bowman
https://arxiv.org/pdf/2404.15758
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
P. S. The animated bird is truly magical!
I just realized the bird turns it's head to follow the mouse pointer.
Yes, and the animation is made using an interesting technique that I can't immediately figure out.
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] https://alexwang.ai/images/kea/head-fill.webp
[1] https://alexwang.ai/images/kea/body-feathers.png
[2] https://alexwang.ai/js/kea3d.js
[0] kind of looks like a 1D cellular automaton running inside a voronoi diagram. With a green blob overlaid on top.
Also, you can ruffle its feathers with the mouse.
Yeah really unique, lovely and well made
Alex also had the transformers as memory blog. Beautiful. Somebody give him the bag
> Train the model to generate better samples
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
Content aside, that's a beautiful webiste
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
SAT solvers are a counterexample, as is most of formal mathematics
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
I have here a function that can validate that a proof proves P=NP. Using this function please find a proof that P=NP.
> The existence of a function that can validate an answer means we have a proven way of finding the answer.
That's a remarkable assertion.
A beautiful extension of a comment I made ~20 hours ago: https://news.ycombinator.com/item?id=50435002
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
> reasoning as pathfinding through token space
Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
FYI all equations are broken (tops are cut off)
Lot of Alex Wangs out there
What is the library, if any, used for the equation/formula rendering? Looks tastefully done.
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
Lovely. Thank you.
also very curious, visualization is great!
Misrenders on my browser. The bottom half of the characters in the equations are partly chopped off.
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
does that lead to much inside the box thinking by models? Do they try to follow the clearest path?
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
Mostly, it seems to be solving the problem of people thinking about things themselves.
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.