There’s an intuitive but simplified picture of reasoning: let a model think for longer, and it’ll be more likely to be correct. This aligns well with our human intuition: spend more time and effort on thinking and it’s more likely that we’ll arrive at a correct answer.
But this picture doesn’t seem to explain everything. During RL training, models learn to write longer chains of thought, and they get more accurate as they do. Yet if you take a trained model and force any one chain to run longer, it helps for a while, then stops helping.1 forced a model to keep thinking by appending “Wait” whenever it tried to stop. Accuracy rose at first, and then flattened. Sometimes the answer even gets worse, as if it’s overthinking.2 A neat example is which found tasks where longer reasoning resulted in worse accuracy. If a chain of thought were truly converging toward some optimum, more steps shouldn’t make it worse. So is it really the length of each chain that helps at inference time? Or is it something more subtle?
This post is my attempt to clarify this confusion (mostly for myself) by looking at reasoning as pathfinding through token space. We’ll see that RL maximizes the probability of finding a path from
Reasoning as pathfinding#
Ask a reasoning model the same math problem ten times and you’ll get ten different chains of thought. Rather than thinking of reasoning as one mysterious process heading toward an answer, let’s think of it as pathfinding in token space. There are many paths from the problem to a correct answer, and the model traverses one of them with each sample.
Consider the problem: “what’s
We can think about reasoning as following this general pattern. We are given a starting point, the prompt, which we’ll call
So given a starting point
The figure at the top of this post sketches this picture. Each gray line is a sampled path: it starts from the prompt on the left, meanders in token space by sampling
Since many paths land in
The second factor is just 1 if the path lands in
We’ll call
Note that there are two ways of increasing this probability. Either the model can sample multiple paths to the same answer, each with a small probability, or the model can sample a single path to the answer with a very high probability. The success probability doesn’t care which one we get, but in practice we might want to prioritize specific paths or ensure diversity in the samples3 We might revisit this in a future post. .
How do we boost our success probability?#
Best-of- sampling#
The simplest way to pathfind is to have the model
Since we assumed we had a verifier, we can do best-of-
This is a guaranteed axis of compute-scaling. As long as
Train the model to generate better samples#
So far the model was fixed, so
Here’s a simple way to do this. Collect many prompts whose answers we can check. For each one, sample some paths from the model. We keep the paths that land in
RL with a verifiable reward, which is how today’s reasoning models are trained, turns out to do almost the same thing. Let
meaning the expected binary reward is exactly the success probability
Computing the gradient of the success probability via the log-derivative trick, we arrive at
which is exactly the policy gradient, also called the REINFORCE estimator.
Now estimate this gradient with
This gradient involves two terms: the empirical success rate
In other words, a step of REINFORCE with a binary reward is a step of rejection-sampling fine-tuning on the model’s own sampled paths, scaled by the current success rate
This also explains the first observation from the intro, that chains of thought get longer during RL training. Each step pulls the model toward the paths that landed, so if the paths that succeeded tended to be longer than the ones that failed (e.g. because they include backtracking behavior, gather more feedback from the environment, or just dumb luck), the model will learn to write longer paths.
Amortization and generalization#
During RL training, we spend a lot of compute to search for good paths and then we train the model to generate those paths directly. Once trained, the model can produce the answer without spending all this extra compute on sampling multiple paths. This is called amortizing the search: we pay for it once during training, so we don’t have to pay for it every time we ask the model a question.4 This idea of using expensive search to find a better answer and then training a network to output the answer cheaply, shows up outside language models too. runs a tree search at every move of self-play and trains its policy network to prefer the moves that were found ( is the same idea). Physicists and weather forecasters do it with simulators: they run an expensive hand-crafted simulation to get an answer, then train a neural network to predict that answer directly. Since we train a single model on many prompts, the model can potentially generalize to new prompts it has never seen.
The figure below shows this for four prompts, each with its own
Revisiting some observations about reasoning#
Once we see a reasoning model as a distribution trained to land in
Chains of thought are not guaranteed to be faithful#
Our verifier above only checks whether the answer is correct. The reward is 1 if
If we want meaningful chains of thought, we need to adjust the rewards or the training objective.
Reward hacking comes from both the verifier and the sampler#
We also assumed the verifier is never wrong, that
Thus reward-hacking comes from an imperfect verifier, which fails to capture what we truly want, as well as an imperfect sampling procedure which fails to stay away from these shortcut regions.
In the figure below, the dashed circle is the set of correct answers and the red ellipse is
This visualization also shows why scaling up RL (generating more samples and longer samples) makes reward hacking more likely. The more extensively we “fuzz” the verifier, the more likely we’ll find a path that lands in the misspecified region of
So what objective is reasoning solving?#
Reasoning isn’t actually solving any objective at all. A path is not guaranteed to converge to the correct answer. It’s only in aggregate, over the distribution of paths, do we get a high probability of success.
RL training directly raises the success probability
Muennighoff et al. 2025 forced a model to keep thinking by appending “Wait” whenever it tried to stop. Accuracy rose at first, and then flattened. ↩︎
A neat example is Gema et al. 2025 which found tasks where longer reasoning resulted in worse accuracy. ↩︎
We might revisit this in a future post. ↩︎
This idea of using expensive search to find a better answer and then training a network to output the answer cheaply, shows up outside language models too. AlphaZero runs a tree search at every move of self-play and trains its policy network to prefer the moves that were found (expert iteration is the same idea). Physicists and weather forecasters do it with simulators: they run an expensive hand-crafted simulation to get an answer, then train a neural network to predict that answer directly. ↩︎