Article

Conversation

On SFT, RL, and on-policy distillation

Why the SFT-RL pipeline works, where on-policy distillation fits, and how self-distillation goes wrong.
Authors: Will Brown & Claude Opus 4.7 
April 30, 2026

[Editor's Note: Arguments are mine, writing is Claude's. This is partially an experiment in trying to get Claude to help speed-write and structure technical research blogs, drafted initially as artifacts and refined via "debate". I have too many blog ideas that I never get around to writing up, but the models finally feel good enough to help out with this (hopefully -- let me know what you think).]
§1 — The standard pipeline and the compounding argument
Most post-training pipelines are some version of "SFT first, then RL" — pre-train, supervised-finetune to get a baseline, then run RL once SFT data dries up or stops moving the needle. People usually treat this ordering as a convention, but there's a real argument for it that's worth being explicit about. The argument is about which sampling distribution your method gets to compound with, and where the resulting performance ceiling sits.
When I say SFT in this post I mean teacher SFT: training on completions produced by some teacher model. (Plain instruction-tuning on human-curated data is the same shape, with humans as the teacher.) The defining property is that the sampling distribution is fixed at dataset-construction time. As the model improves during training, the data does not. Once the student gets close to the teacher's distribution, more SFT mostly memorizes — the marginal example is no longer informative. SFT's ceiling is roughly the teacher's.
RL is the opposite. The student samples its own rollouts, the gradient updates the policy, and the next batch is sampled from the improved policy. Improvements compound back into the sampling distribution. The ceiling isn't determined by the data — it's determined by whatever the verifier can grade.
This produces a tipping point. When current performance is far below the teacher and teacher data is cheap, SFT bits are extremely cheap-per-improvement — you're learning capabilities you don't have, from a source that does. As current performance approaches the teacher, marginal SFT examples get less informative, and the student's own rollouts start producing genuinely new strategies via lucky exploration that an RL gradient can extract. Past the tipping point, rollout compute is better spent on RL than on more SFT.
Rejection-sampled SFT (SFT-RS, sometimes RFT — sample from the teacher or the student, filter for correctness, train on the survivors) is strictly better than vanilla teacher SFT in expectation, but it doesn't fix the compounding problem. The sampling distribution is still pinned to whatever you're filtering. Once your filter saturates — everything correct is included, everything wrong is discarded — you've hit the same ceiling. SFT-RS shifts the curve up, but it doesn't change its shape.
None of this is new, but it's worth saying clearly because everything else in this post is about how to push past that ceiling, and the available moves depend on what you have access to.
§2 — Same-family vs. different-family teachers
"Doing SFT" involves choosing a teacher, and the teacher's relationship to your student is a major axis of efficiency that doesn't get enough attention.
Same-family teacher means tokenizer-matched and recipe-matched: same vocabulary, same general training pipeline, ideally the same base model at different scales. Qwen3-32B teaching Qwen3-8B-Base is the canonical example, originating in the Qwen3 technical report and revisited prominently in Thinking Machines' OPD post. The teacher's outputs sit in a distribution that's structurally close to what the student naturally produces. The SFT signal — for each token, "the teacher would have said this" — is mostly about the capability gap, not stylistic differences. Per-token logprobs from the teacher are directly comparable to the student's, because the vocabulary is shared and similar training recipes produce similar distributional shapes.
Different-family teacher means tokenizer mismatch, recipe mismatch, or both. Distilling a frontier closed model into an open base, or moving across model families. Two specific costs:
Tokenizer mismatch means every teacher completion has to be re-tokenized in the student's vocabulary. You lose information at boundaries, and per-token logprobs no longer correspond to "the same prediction at the same position" between models. Soft-target distillation is essentially impossible without expensive workarounds.
Recipe mismatch means the teacher's outputs carry stylistic and structural artifacts — how it formats lists, when it uses chain-of-thought, what register it speaks in — that are pipeline byproducts, not capabilities. The student has to absorb those alongside the actual content. A nontrivial fraction of the SFT bits go toward learning surface form rather than competence, and the resulting student often looks like a slightly worse version of the teacher's style without having internalized the teacher's reasoning.
Most cross-family distillation in practice loses something in the gap between "what the teacher knows" and "what the student can absorb without learning the teacher's pipeline." Same-family SFT is much more efficient per unit of capability transferred, even if you can't always afford to limit yourself to it.
This matters because the next move on the algorithm tree — on-policy distillation — is essentially only available in the same-family setting, and that constraint shapes the rest of this post.
§3 — On-policy distillation as the same-family upgrade
On-policy distillation (OPD, as in Lu et al. 2025 and the Qwen3 technical report before it) is the obvious move once you've laid out the previous two sections. The student samples its own rollouts — so you get RL's compounding in the sampling distribution — but each token in the rollout is graded by the teacher via per-token reverse KL:
∇θJOPD(θ)=Ex, y^∼πθ(⋅∣x) ⁣[∑t(log⁡πT(y^t∣y^<t)−log⁡πθ(y^t∣y^<t)) ∇θlog⁡πθ(y^t∣y^<t)]\nabla_\theta J_{\text{OPD}}(\theta) = \mathbb{E}_{x,\, \hat{y} \sim \pi_\theta(\cdot \mid x)} \!\left[\sum_{t} \bigl(\log \pi_T(\hat{y}_t \mid \hat{y}_{<t}) - \log \pi_\theta(\hat{y}_t \mid \hat{y}_{<t})\bigr)\, \nabla_\theta \log \pi_\theta(\hat{y}_t \mid \hat{y}_{<t})\right]

The advantage at each token is "how much the teacher prefers this token relative to the student." Dense, on-policy, reverse-KL. The reported numbers are striking — roughly 9–30× less compute than RL on AIME-style benchmarks, with the gap widening when teacher logprob calls can be parallelized.
This raises a question that figure 1 implicitly answers: why does OPD's practical ceiling sit above SFT-RS's, when both target the same teacher? Cheaper sampling is part of it — the teacher only forward-passes over student tokens, which is essentially prefill, and prefill is much cheaper than generation. But that mostly buys faster convergence, not a higher ceiling. The deeper reason is on-policy state coverage. SFT and SFT-RS train under the teacher's state distribution but evaluate the student under its own; the exposure-bias gap that opens up over long rollouts caps off-policy practical performance somewhere short of "teacher quality" on the actual evaluation distribution. OPD trains on student rollouts, so the gap doesn't open. Same nominal asymptotic target, different practical ceiling.
The catch is that OPD requires a same-family teacher, and the dependence is harder than people often acknowledge. You need tokenizer match just to compute the loss — the per-token KL is between teacher and student distributions over the same token positions. You also need at least approximate recipe match, because the per-token signal needs to actually be informative on the things you care about. If the teacher and student were trained with very different recipes, the reverse-KL gradient is dominated by "the teacher would have phrased this differently" rather than "the teacher would have reasoned differently here."
So OPD is the "best of both worlds" corner only when you have a same-family teacher available. When you do, it tends to dominate at moderate compute budgets — do mid-train SFT to get into the teacher's rough neighborhood, then OPD instead of RL to approach the teacher's level efficiently (you get to explore with a smaller model, and prefill is cheap). The ceiling is a different question. OPD targets the teacher's distribution by construction, so in the limit it's bounded by the teacher; RL's ceiling is bounded by the verifier, which can be much higher. So the framing isn't "OPD beats RL," it's "OPD gets you to the teacher's level much faster than RL would, and most of the time that's where you wanted to be anyway." When you don't have a same-family teacher, you fall back to either cross-family SFT (eat the recipe-mismatch tax) or RL (eat the sparsity tax) — or, more recently and speculatively, to self-distillation.
§4 — When you don't have a same-family teacher
The natural response to "OPD is great if you have a same-family teacher" is "what if I don't?" Self-distillation tries to answer this by using the student itself as the teacher, with privileged information in the teacher's context that the student doesn't see at sampling time. Two instantiations have shipped recently; they share the same algorithmic shape and differ only in what privileged info goes to the teacher.
SDFT (Shenfeld et al. 2026) conditions the teacher on an expert demonstration — a worked example, possibly for a different task. Student samples a trajectory without the demonstration in context; teacher computes per-token logprobs over the trajectory while seeing the demonstration; student updates toward teacher via reverse KL. Fully on-policy. The demonstration provides distributional pull without leaking the answer, and the paper makes the assumption underpinning this explicit (more on that in §8).
OPSD (Zhao et al. 2026) conditions the teacher on the ground-truth answer instead. Same setup otherwise. The teacher knows where the trajectory is supposed to end up; the student doesn't. This produces a much sharper distributional shift in the teacher than a demonstration does.
In both cases tokenizer match is automatic (same model). Recipe match is automatic (same model). What you trade for that is that the privileged-info conditioning shifts the teacher's distribution — modestly for SDFT, more aggressively for OPSD — away from the student's natural distribution.
Both sit at exactly the same dial settings as OPD. The only difference is the choice of teacher. That difference turns out to be where the failure mode lives.
§5 — The geometry of gradients
To see why, it's worth working through what each method's gradient actually looks like in parameter space. The shapes are different in instructive ways.
RL: sparse, but saved by destructive interference
The standard complaint about RL is one bit per episode — a binary "you got it right" spread across thousands of tokens. The framing I find more useful: the apparent sparsity is what keeps RL's gradient updates honest.
In a GRPO-style step, every token in every rollout gets an advantage. Within a group, the advantages have approximately zero mean by construction (subtract the group baseline) and some variance. Each per-token advantage gets multiplied by the gradient of the log-probability of that token, producing a vector in parameter space. The total update is the average over the batch.
Most of those vectors are noise. Reward is sparse and broadcast-assigned; most tokens didn't actually contribute to whether the answer was right. Their advantages are nonzero only because they happened to share a trajectory with a reward, not because they were causally responsible for it. So the batch contains a swarm of small, mostly-random parameter-space vectors, with a small consistent bias along whichever dimensions actually correlate with reward ("think longer," "double-check arithmetic").
The reason large-batch, low-learning-rate RL is robust is not that each per-token gradient is informative — most aren't — but that the uninformative ones cancel. What survives the average is the small consistent component along the directions that actually correlate with reward. There's some empirical support for this: RL updates have been observed to be sparse in parameter space, modifying small subnetworks. That seems like the destructive-interference picture cashing out.
It also explains why RL feels "safe but slow." Each step moves the model a tiny amount in a direction you can trust. The sparsity isn't purely a bug; it's the price for an unbiased estimator.
SFT: dense, biased, but spread out
SFT does the opposite. Every token gets a one-hot label. The gradient density is enormous — one informative update per token, no broadcasting required.
But the SFT gradient distribution does not have zero mean. By construction, it's biased toward the data distribution. There's no analog of advantages cancelling. Constructive interference, not destructive.
So why doesn't SFT blow up? Two reasons. First, the data distribution is itself diverse: the bias points in many slightly different directions across many examples, so you get a soft principal-components decomposition where the model drifts toward the data manifold as a whole rather than any one example. Second, SFT is diffuse. Most of what an SFT step is doing is reinforcing things the model already half-knew. There aren't many tokens where the data is asking the model to put large mass on something it currently treats as near-zero probability.
This is why SFT is forgiving. The data can be a bit messy, the learning rate can be a bit off, and you mostly drift. The bias is real, but it's unconcentrated.
OPSD: dense, biased, and concentrated
OPSD's gradient has a different shape from either of the above, and the difference is instructive — it points at why pure self-distillation isn't quite the final answer.
Consider a long math rollout where the student gets the answer wrong because, somewhere in the chain of thought, it failed to make a key observation — pick the right substitution, notice a trick. Call that token (or short span) the pivot token. For the student, the probability of producing the pivot token might be very low — say 0.01. For the teacher, conditioned on the answer, the probability of that same pivot token is much higher — say 0.6, because once you know where the solution is going, the right substitution becomes obvious.
The per-token reverse KL between "student says 0.01" and "teacher says 0.6" is roughly log(0.6/0.01) ~= 4.1. For a typical token where both put around 0.3, the contribution is essentially zero. So one pivot token contributes on the order of a hundred times more to the loss than a typical one. The gradient is dominated by it.
What does that gradient do? It pushes the student's log-probability of the pivot token up sharply, given the prefix. The corresponding direction in parameter space is "make this rare token less rare in this context." Critically, this update is not being averaged against many other vectors pointing different directions. RL's saving grace was that noise vectors cancelled. SFT's saving grace was that bias was diffuse across many tokens that the student already supports. OPSD has neither. One concentrated tug, in one step, toward a region the model didn't previously believe in.
OPSD ships with per-token point-wise KL clipping — cap the per-vocabulary-entry divergence contribution at each position so that, in their words, "a small subset of stylistic tokens [doesn't] dominate the training signal." Without it, the paper reports performance collapse within ~100 steps. That's figure 4 cashing out. The fix works, and what it tells us is direct: the KL signal in self-with-hint distillation is concentrated enough that you have to budget it. The next move, then, is to look for a teacher whose KL is naturally diffuse, rather than relying on clipping to make a concentrated one tolerable.
§6 — The sparse/dense × biased/unbiased family
The gradient analysis above places each method in a small taxonomy. The two main axes are sparse vs. dense (does each token get a signal, or just the trajectory) and biased vs. unbiased (does the expected gradient point in a fixed direction relative to the student, or only along the directions reward correlates with).
SFT, OPD, and OPSD all live in the "dense, biased" cell. They're distinguished by where the bias points (toward the data, toward a same-family teacher, toward self-with-hint) and, more importantly for the failure-mode analysis, by how that bias is distributed across tokens. That's the third axis — concentration:
This is the punchline of the geometric analysis. RL is sparse but unbiased. SFT is dense and biased but diffuse. Same-family OPD inherits SFT's diffuseness because the teacher distribution is calibrated to the student's family. OPSD is the unique case where you get density, bias, and concentration at once — which is why it ships with explicit defenses (KL clipping, fixing the teacher to the initial policy) that the other methods don't need.
§7 — An illustrative meta-algorithm
All four methods are special cases of a single token-level policy gradient:
with two scalar knobs and a teacher-policy choice:
α ∈ [0,1] — how on-policy the sampling distribution is.
λ ∈ [0,1] — how much of the per-token advantage comes from a teacher KL versus a sequence-level outcome reward.
π_T — the teacher policy: which model, conditioned on what context c_T.
Once you fix α and λ, almost everything that matters — bias, concentration, stability — is a function of how π_T differs from π_θ at the per-token level on the student's rollouts. The interior of this space — α, λ ∈ (0,1), or mixed teacher choices — is mostly unexplored, and seems like where the next round of progress lives.
A few things drop out:
SFT is "distillation from a degenerate teacher": letting π_T(y|x) = δ_{y_data}(y) recovers cross-entropy up to constants. The reason SFT is safe is not that this teacher is bad — it's that it's averaged across many varied examples, which makes the bias diffuse.
RL is "no teacher": set λ=0 and the per-token signal collapses to broadcast outcome reward. The destructive-interference story follows from this.
OPSD has identical dial settings to OPD: the only difference is π_T. That clarifies why it can match OPD when it works (same algorithm) and why it's where the failure mode is sharpest (most distributionally aggressive choice of teacher).
A caveat about what this factorization is for: it’s illustrative of the kind of algorithm conceptually aligned with the Pareto / SFT-RL phase-change picture — not a recommendation to mix sources at intermediate (α, λ). I’m not sure that’s even a useful algorithm. The clean corners (SFT, RL, OPD) are where the statistics work out without importance-sampling corrections, and they correspond to qualitatively different regimes that differ mostly in the KL budget β. The interesting axis of variation across them is β; interpolating α and λ is a tangent.
The practically-interesting structure looks more like “pick β, then find the best teacher for that β.” The meta-algorithm above makes the corners visible, but the actually hard problem is the inner one — and teacher optimization isn’t clean or cheap. It’s typically discrete (which model? which prompt? which hint?) and doesn’t decompose neatly into the gradient framing. The Pareto picture in §8 is where that question actually lives.
§8 — Toward an optimal teacher
The geometry analysis turns the optimal-teacher question into an optimization problem. We want a teacher policy π_T that produces a large reward improvement on the student per step, subject to a hard KL constraint that keeps updates stable. In Lagrangian form:
This traces a Pareto curve as β varies. Different methods are different points on it.
The SDFT paper's framing is worth engaging with directly here, because it offers a candidate route to the curve's interior that doesn't quite work for the reasons we want it to. The paper formalizes the ICL assumption — that a model conditioned on an expert demonstration approximates the optimal policy for the task — and conditions its analysis on the availability of such demonstrations. That assumption is doing a lot of work. Practically, this gives a similar ceiling (conceptually) to SFT and SFT-RS: you're only as good as the demonstrations, and you can't bootstrap beyond them.
We'd rather be in a world with curves more like RL, where the ceiling is only a function of our ability to verify, even if we can't always solve (without extremely expensive sampling). We can make weaker assumptions about the ability to verify (which we need anyway if we want to measure performance) plus mild regularity conditions on the distribution: that performance in accuracy degrades gracefully across the task distribution in a smooth latent space coupling task characteristics to the model's solve likelihood for a given checkpoint, and that updates should be net-positive-EV in distribution solve rate when sufficiently well-regularized — the high-batch-size, low-learning-rate, unbiased limit of, e.g., infinite-batch RL. This is what's needed for compounding gains where the only ceiling is the grader's performance, with the question being how to get there faster while retaining stability.
Policy gradient — at least within outcome-reward GAE — is roughly a minimum-variance unbiased estimator for the gradient of expected reward. The aim is to be able to turn a knob that trades bias for stable-yet-aggressive updates. OPD and SFT are two points on this curve, but both need a real teacher. Using a bigger smarter model improves the teacher but isn't directly workable for OPD without tokenizer match. There are cross-tokenizer methods, but ultimately we want something that lets us sweep teacher performance versus bias cleanly and cost-effectively.
Several plausible approaches target this objective at different levels of the stack:
Per-task prompt optimization over the Lagrangian. Use something like GEPA over the Lagrangian objective on a per-task basis, with teacher sampling to estimate rewards. The optimizer searches the space of prompts (hints, conditioning) for the one that maximizes E[Δreward] − β·KL on the current student. This is a cheap inner loop that can produce a usable teacher without retraining anything.
Distribution-level prompt optimization. Train a model to write good minimal hints when given a "bad" hint — whitebox RF access, the answer itself, a known-good demonstration. The training signal is the same Lagrangian, aggregated across the task distribution. The output is a hint-rewriter that turns large privileged-info hints into small ones that move the teacher distribution as little as possible while still improving reward.
Self-prompt-optimization online RL. Treat hints as rollouts in a parallel environment that improves alongside the main policy via an RL objective, with loss components balanced adaptively (e.g., a smoothed minmax over reward-delta and KL terms). The hint-writer and student co-evolve.
Train a hint-writing model directly with RL. Use correctness-delta × (1 − KL-delta) as the objective, scoped to a specific model / distribution / task / "in general" (the same way scopes work for judge-model training). The hint-writer is a separate artifact you can ship.
Borrow from "expert RL + OPD." Some recent models (e.g. DeepSeek V4) layer a teacher signal on top of locally-optimal RL — an analog of the meta-algorithm with both a teacher KL and an outcome reward live simultaneously. The lessons from that line are directly relevant here.
What unifies these is that they all target the Lagrangian objective without requiring a fixed external teacher policy: the "teacher" is constructed (per task, per distribution, or online) to be locally optimal for the current student — high reward, low KL, surgical rather than broadcast.
In the limit, RL might still be optimal in the infinite-compute regime for the hardest heavy-tail problem distributions — the cases where any teacher distribution adds bias the student would've eventually corrected anyway. But it seems quite plausible that there's a nice meta-algorithm for interpolating more cleanly between distillation and RL: without needing a real teacher, and with compute-optimal learning at each point along the curve. Working that out is the open question I find most interesting in this space.

References 
· Lu, K. & Thinking Machines Lab. On-Policy Distillation (2025). 
· Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD, 2026). 
· Shenfeld, I. et al. Self-Distillation Enables Continual Learning  (2026). 
· Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023). 
· Mukherjee, A. et al. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (2025). 
· Qwen Team. Qwen3 Technical Report (2025). 
· Ross, S. et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAGGER, 2010).

Artifact: https://claude.ai/public/artifacts/478fd2c4-e37d-4f1e-8295-780d811b1859

Markdown: https://claude.ai/public/artifacts/e03bf1a9-a6ab-4324-8ed2-cca1543f0878

2:23 AM · May 1, 2026

479.8K

Views

View quotes

Post your reply

xlr8harder

@xlr8harder

May 1

What about reasoning trace quality under SD? It seems like the reasoning traces will be much less valuable: referring to examples that the student can't see implicitly or explicitly. Seems like it would harm performance/induce hallucinations. Is it a valid concern?

absolutely -- roughly the main issue here. reasoning traces need to be "natural" for the student (= low KL mismatch)

Great post. In terms of Claude writing - I recommend using this to make it more enjoyable to read. github.com/blader/humaniz My flow to write is: - random notes/thoughts - feel like there are enough of those for a post - put it all together in Claude - discuss the overall idea

humanizer/SKILL.md at main · blader/humanizer

i'm just using claude.ai in web / mobile, writing to html or markdown artifacts. works great for me

This very surprisingly doesn’t feel like reading from Claude. I wonder if it’s a marked improvement from 4.7 or if I just haven’t gotten hyper aware of 4.7 mannerisms yet. Or maybe you have some really good prompts.

will brown

@willccbb

May 1

long rambling voice note -> artifact, then 10-15 rounds of giving corrections on-policy self-distillation, essentially

can't you make a similar interpolation for two completely unrelated loss functions? how is that a meta algorithm

the real meta-algo is more S8's "distill from the reward-optimal teacher in some KL radius" RL is the KL -> 0 limit of this, all others are heuristically justifying optimality vs closeness via other extremal settings

nice article, for next time: would be nice to have a markdown format gist or something so i could more easily discuss with claude about your ideas and his writing lol

gotchu

SFT, RL, and On-Policy Distillation: A Technical Deep Dive

I still find it feels like a step up from 4.7, but I agree with you that some of the responses can feel like 'slop' - I've noticed it's especially noticeable with more abstract topics.

matteo saponati

@matteosaponati

May 29

fantastic write-up! we find that OPSD is actually pretty predictable given its very own nature. one can predict the final performance from the initial student-teacher gap. x.com/matteosaponati then you can try different OPSD configurations and see which ones fits better in

Quote

matteo saponati

@matteosaponati

May 29

A PREDICTIVE LAW FOR ON-POLICY SELF-DISTILLATION I am happy to share the second bit of my work at @tufalabs On-policy self-distillation (OPSD) uses rich, arbitrary feedback as a learning signal, but its reliability remains unclear. A method to predict OPSD effectiveness before

@nrehiew_

May 1

Nice article! Thought Fig3 in particular was really cool. I have always thought of them in terms of distribution: - SFT: Fixed target distribution. Distance from initial 'policy' distribution dependent on SFT data source/teacher -Rejection Filtering: Similar to SFT but

Love this. RE format I think the right way in the future will be that instead of filling info with text, we’ll distribute the facts and each user’s LLM will help teach them the material corresponding to their level. This’ll make it less of a maybe slop blog and even more

distil or die

GIF

874

@init_malachi

May 1

this was helpful and clear in your intent

DAgger and distillation is still understudied especially in this long-horizon era

or just forget about privileged ICL teachers, *train* privileged critics (using GAE targets). After all, youre just after that dense signal

Quote

Xiuyu Li

@xiuyu_l

Apr 24

DSv4 co-evolves the policy model and the judge. Exciting to see this progress! We explored a similar idea in V1-PairRL (harmandotpy.github.io/v1-verification). Check it out if you’d like to learn more about this direction :)

Thanks for sharing! I'll be reading through this very carefully

Appreciated the comparisons a lot. One of the better articles I’ve read so far this year.

Instant classic.

thanks for sharing, really interesting

323

@llllvvuu

May 1

I have similar thoughts on what’s sus about self-distillation and how to fix it. The one that put the answer directly in the prompt made me wince the most. That is what STaR did in cases where rejection sampling was not successful but it’s hard to imagine that producing good CoT

762

Thiyagarajan Maruthavanan (Rajan)

@mtrajan

May 2

§7 nails it. Every method is the same algorithm, only the teacher changes. Which makes §8 the real question. And honestly, the Environments Hub is already half the answer. Most people just haven’t connected the dots yet. Teacher manufacturing is the bottleneck for the next five

Rob Haisfield

@RobertHaisfield

May 4

Great blog, super informative. One thing I felt was missing though… I know the answer is sort of mixed in throughout the post, but simply, why is it so important for your methods to be on-policy?

we have recently added these methods in TRL if you want to try them out huggingface.co/docs/trl/sdft_ huggingface.co/docs/trl/sdft_ and also huggingface.co/docs/trl/ssd_t

I will release a self-distillation work soon, pure sampling w/o RL can beat GRPO using pass-1, please stay tuned

thanks for the playbook

I don’t care claudeslop not claudeslop this is awesome thanks for posting it

Was this on your radar?

arxiv.org

A Survey of On-Policy Distillation for Large Language Models

As Large Language Models (LLMs) continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become a central engineering problem, and...

unroll

Elon Rusk , Hardcore AI Engineer.

@ElonRusk314

May 1

> The ceiling isn't determined by the data — it's determined by whatever the verifier can grade. The ceiling is also determined with the number of <environments,prompt> tuples you've, in addition to verifier.

Thanks for sharing!

Hey will. Great post. What are thoughts on RLSD and SD-Zero?

calling this claudeslop would be a disservice. this was very knowledge-"dense". we need more of this.

looks like a lot of math to explain why we keep reinventing the wheel instead of just asking if rl is actually necessary

Claude co-authoring a distillation paper is peak recursion. On-policy framing explains why merges are either magic or garbage.

Good post! One thought: could using hint+prompt methods bias the student toward actor-like behavior rather than true planning?Intuitively, since the hint is reflected in the teacher’s per-token log-probs, might the student overfit to following the hint instead of planning ability

Garrett Goon

@GoonGarrett

May 1

Very good. Claude's writing was entirely fine, would be happy to see more

Really enjoyed this distributional view of SFT/RL/OPD and the caution on self-distillation. Our LLM post-training survey takes a related lens: off-policy support expansion, on-policy policy reshaping, and behavioral consolidation. Sharing in case useful:

arxiv.org

Large Language Model Post-Training: A Unified View of Off-Policy...

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference...

Nagendra | OmniMind Labs

@Nagendra27n

May 3

@karpathy

Built a free local AI assistant inspired by the idea that AI should be private and accessible to everyone. No API keys, runs offline, MIT licensed.

GitHub - nagendra27na/openassistant: Free local AI assistant — no API keys, no cloud, no cost....

At some point I feel sad because we do not have access to the logits from the closed-source model, sometimes not even the full trajectory. Like researchers are building better algorithms to utilize the dense signal...

Hi Will, it's a super cool blog! I love and agree with most of the insights, especially on viewing the teacher in on-policy distillation as an optimization problem. But I have one question: why is the concrete and biased gradient in OPSD an issue? Can we just increase the batch

306

Chidhambararajan R (a.k.a Chidha)

@TheSeriousProg

May 1

Hey

@willccbb

I made an article with visualizations of a grpo training. Please let me know your thoughts on it !! I made a detailed experiment visualization inspired by this article, also tries to visualize the precision mismatch problem

Quote

Chidhambararajan R (a.k.a Chidha)

@TheSeriousProg

May 1

Visualizing and Demistifying Reinforcement Learning (GRPO)

An attempt at visualizing reinforcement Learning with GRPO algorithm during training. The algorithm which powers the modern intelligence race There are many algorithms under Reinforcement Learning,...

, just having time to read your article. I realize with the dream algo you aim (keeping sample efficiency of OPD, local optimal update with KL constraints, rewards/verifier from RL), perhaps bringing value function back can be an idea?

Bhavya Goyal

@777BHAVYA

May 1

what do u think about the merging of model trained for diff tasks can that help continual learinig in some way want to know your opinion

280

To view keyboard shortcuts, press question markView keyboard shortcuts

Article

Conversation

§1 — The standard pipeline and the compounding argument

§2 — Same-family vs. different-family teachers

§3 — On-policy distillation as the same-family upgrade

§4 — When you don't have a same-family teacher

§5 — The geometry of gradients

§6 — The sparse/dense × biased/unbiased family

§7 — An illustrative meta-algorithm

§8 — Toward an optimal teacher

To view keyboard shortcuts, press question mark
View keyboard shortcuts