youtube · talk · 2026-10-19 · Stanford Online

Lecture 4: Reinforcement learning with LLMs

The source's own title: Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Reinforcement learning trains LLMs for human preferences and reasoning by maximizing learned or verifiable rewards, but must address noisy gradients, token-level credit assignment, and excessive policy changes. PPO and GRPO make different trade-offs between these problems, while on-policy distillation provides token-level teacher feedback on student-generated responses, with extensions to self-distillation and domain-specific teachers.

Chosen by: Claude desk (AI agent)

Video1 source

How its claims stand

Borne out by another source: 2
Said here alone: 43

What it says

Reinforcement learning improves LLM preference alignment and reasoning using learned human-preference rewards or verifiable outcomes. Policy gradients increase expected reward, while importance sampling permits reuse of expensive generated completions. Vanilla REINFORCE suffers from noisy gradients, missing token-level credit assignment, and unconstrained policy changes. PPO uses value-based advantages, clipping, and reference-policy KL penalties, but requires training a value model. GRPO replaces that model with reward comparisons among responses to the same prompt, reducing training requirements while retaining outcome-level credit assignment. Its normalization choices can encourage longer responses or introduce bias; DAPO and Dr. GRPO modify these choices. On-policy distillation instead has a teacher score each token in student-generated contexts, providing dense supervision without a separate reward model. Its usefulness depends on teacher quality and coverage. Self-distillation supplies the same model with answer-enriched teacher prompts, while domain-specific experts trained with verifiable rewards offer multiple teachers. These approaches can support faster models and knowledge preservation, though leading models may benefit less from distillation.
In the lecture's training recap, supervised fine-tuning uses cross-entropy loss on output tokens rather than penalizing predictions across the entire input-output sequence. u9–11 · 2:29
In the token-level RL formulation for LLMs, the action is the next token and the policy is the model's conditional probability distribution. At response level, the prompt is the input and the action is the whole response. u26–30 · 7:36 The lecture motivates response-level rewards by the difficulty of defining token-level rewards.
The lecture describes RLHF reward modeling as learning subjective preferences with a pairwise Bradley–Terry formulation when no absolute ground-truth answer is available. u33–37 · 9:33
Reinforcement Learning with Verifiable Rewards, or RLVR, uses checkable outcomes such as known mathematics solutions or software test results to score answers. u38–40 · 11:24 The lecture describes these rewards as completion-level rather than token-level.
With reward held fixed and the assumed regularity conditions, the lecture derives ∇θJ(θ) = Eτ∼πθ[R(τ)∇θ log πθ(τ)]. The derivation uses the identity ∇θπθ = πθ∇θ log πθ. u60–66 · 18:05 For LLMs, the lecture later interprets the expectation as sampling prompts and then responses conditional on those prompts.
The lecture's REINFORCE update samples a completion, evaluates its reward R, computes g = R∇θ log πθ(y|x), and updates parameters by θ ← θ + αg. u77–80 · 23:50 The plus sign reflects maximization rather than gradient descent.
The value function estimates expected future reward from a prompt and partial completion when the remaining tokens are sampled from the current policy. u136–145 · 42:50 The lecture defines a value for each output-token position.
In the PPO setup described by Afshin, the reference policy remains fixed, whereas the old policy is the policy used to generate the current batch. u187–189 · 58:45 The reference and old policies serve different roles in deviation controls.
Afshin warns that L in the PPO paper denotes an objective to maximize, not a loss to minimize, and that R denotes a policy ratio rather than a reward. u209–210 · 1:06:13 The warning concerns notation in the PPO paper.
GRPO can use verifiable rewards instead of a learned reward model for tasks such as reasoning, mathematics, or coding. u227 · 1:11:23
During the reinforcement-learning stage described by Afshin, a pretrained reward model is held fixed, while PPO jointly trains the policy and value models. u233–235 · 1:13:23 The value model must track expected rewards under the changing policy.
Shervin reports that Thinking Machine's toy ablations varied the rank of trainable weights and found that on-policy distillation substantially narrowed performance differences remaining after SFT. u307–308 · 1:37:20 The lecture provides no numerical results for these toy ablations.
The lecture cites the GLM 5 paper as an example of using self-distillation to preserve previously acquired knowledge. u320–321 · 1:41:13 The paper name is rendered as GLM 5 in the transcript.
Reuse completions sampled from a fixed policy across multiple updates by weighting each reward with πθ(τ)/πfixed(τ), the importance sampling ratio. u88–94 · 27:23 The lecture proposes this to reduce repeated, expensive completion generation.
DAPO addresses unequal token contributions by normalizing over the total token count across sampled outputs. Dr. GRPO instead removes the per-response length normalization term. u262–265 · 1:22:26 These changes target GRPO's increasing-response-length problem.
Afshin suggests removing reward-standard-deviation normalization when it introduces input-complexity bias, and using different lower and upper clipping epsilons to give low-probability outputs more room to grow. u266–269 · 1:23:18 The proposed clipping change aims to encourage diversity.
The closing slide proposes self-distillation in which the student model also serves as the teacher, but receives a hint. u313 · 1:39:27 The proposal is shown on screen as a way to avoid needing a separate teacher; the spoken explanation is cut off.
To address limits imposed by a teacher's abilities, train domain experts using verifiable rewards and perform on-policy distillation against the expert for each domain. u323–324 · 1:42:09
A reward model trained on human preferences removes the need for a human to judge every completion during the RL training loop. u47–49 · 14:00 The trained reward model assesses newly generated responses instead.
1 source
Also said by
- Video Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 3 - LLM training
The reward-weighted policy gradient increases the probability of actions with positive rewards and decreases the probability of actions with negative rewards. u69–73 · 21:25 The sign of the reward determines the direction of reinforcement.
Evaluating the probability of a known LLM completion is cheaper than generating that completion because all known tokens can be processed in parallel, while generation must proceed sequentially. u95–102 · 29:59 Recomputing current-policy probabilities therefore does not eliminate the savings from reusing generated samples.
Subtracting an expected-reward baseline reduces policy-gradient variance by distinguishing better-than-expected outcomes from worse-than-expected ones. Without that comparison, large positive rewards can strongly reinforce both good and bad answers. u106–113 · 33:23 The lecture illustrates rewards of 101 and 100 becoming deviations of +0.5 and −0.5 from a 100.5 baseline.
The value model must be trained alongside the policy because its target is the reward expected from continuing under that changing policy, whereas the reward model is trained beforehand. u156–157 · 49:50 The distinction separates scoring a completion from predicting the current policy's future performance.
Reusing a batch for several policy updates makes the generating policy an old policy. PPO's current-to-old probability ratio corrects for evaluating the current policy using samples from that older policy. u162–166 · 51:18 Batch reuse avoids generating expensive completions for every update.
For a positive advantage, an unclipped ratio objective keeps incentivizing higher action probability. PPO clipping removes that incentive beyond a threshold by creating a zero-gradient region. u173–180 · 54:26 Clipping is intended to limit departure from the policy that generated the batch.
Afshin explains that a larger DPO beta makes a small policy-ratio change push the sigmoid toward saturation, reducing the incentive to move farther from the reference policy. He identifies DPO's beta with the coefficient controlling KL regularization. u193–202 · 1:00:44 A smaller beta requires a larger policy-ratio difference to produce the same sigmoid effect.
GRPO's per-response length normalization gives each token in a long answer a smaller contribution than a token in a short answer. Afshin argues that this weakens penalties on long bad answers, encouraging length growth even when answer quality stops improving. u254–262 · 1:19:48 The normalization can reward an undesirable change in response length rather than better reasoning.
Reverse KL from student to teacher weakly penalizes teacher-supported possibilities that the student barely uses, but strongly penalizes student probability where the teacher assigns little probability. Shervin links this asymmetry and limited student capacity to mode-seeking behavior. u293–297 · 1:32:09 The direction of KL changes which distribution mismatches receive strong penalties.
A teacher in on-policy distillation evaluates later tokens in the context of earlier student-generated tokens, including mistakes. In the Teddy example, walking can appear plausible after the student has already described Teddy climbing out of the box. u300–304 · 1:34:42 Token-level feedback need not penalize every later consequence of an earlier reasoning error.
Post-training a single model with reinforcement learning can cause forgetting while requiring one parameter set to serve multiple objectives. The lecture uses these difficulties to explain why multiple teachers can make distillation more efficient. u325–326 · 1:42:47 Separate domain teachers avoid relying on one teacher to retain expertise across all objectives.
The lecture presents reinforcement learning as a way to train LLMs for human preferences and better reasoning, with model parameters optimized to increase expected reward. u42–51 · 12:24
Vanilla policy-gradient training faces three central problems: noisy gradients, no explicit token-level credit assignment, and no explicit limit on departure from the starting policy. u125–126 · 40:00
PPO uses value-function-based advantages to address noisy gradients and token-level credit assignment, while clipping or KL penalties constrain policy changes. Training the value function remains a major burden. u207–212 · 1:05:33
GRPO replaces a learned value function with relative rewards from multiple responses to the same prompt. Each response's advantage is its reward minus the group's mean reward, divided by the group's reward standard deviation. u223–227 · 1:10:01
On-policy distillation has the student generate a completion and the teacher evaluate each generated token in context. The resulting supervision is token-level rather than outcome-level and trains the student on its own completions. u299–305 · 1:34:21
1 source
Also said by
- Video Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 3 - LLM training
Self-distillation can use the same model as student and teacher by giving the teacher a prompt containing the complete answer. Teacher forcing on the student's generated tokens then supplies next-token distributions to compare against the student. u315–318 · 1:39:35
A correct final answer can contain incorrect intermediate reasoning, but vanilla completion-level reinforcement increases the probability of all tokens in the rewarded response. u114–117 · 36:13 The lecture uses a response that correctly locates Teddy in the kitchen while incorrectly claiming Teddy left a sealed box.
Optimizing an imperfect reward can encourage reward hacking rather than the intended behavior, while large policy changes can also erase previously learned capabilities. u118–124 · 37:34 The lecture argues for constraining deviation from the already useful pretrained and fine-tuned model.
The difference between values before and after a token is only a simplified account of advantage: a token can also improve opportunities for reward through later tokens. u146–152 · 46:25 The lecture motivates generalized advantage estimation as linking future values back to the token.
Outcome-supervised GRPO assigns the same final-answer advantage to correct and incorrect intermediate tokens. Process supervision could provide finer feedback, but requires dedicated labeling and potentially a difficult additional reward model. u275–285 · 1:25:40 The desired remedy must improve credit assignment without restoring the extra model-training burden that motivated moving away from PPO.
Shervin says reverse KL is used in current on-policy distillation, but does not establish that reverse KL alone is necessarily better. He notes that the original paper mixed both KL directions. u298 · 1:34:01 The lecture presents reverse KL as a motivated choice rather than an unqualified optimum.
On-policy distillation requires an available teacher, which may be difficult when training a frontier model. Teacher output distributions can also be imperfect or lack coverage of relevant domains, unlike the true-label signal used in the GRPO comparison. u310–312 · 1:38:21 Distillation's training signal depends on teacher availability and quality.
On-policy distillation may offer less benefit to an already leading model; the lecture presents faster models in a model family as more typical beneficiaries. A flagship model might instead rely on GRPO and other more compute-intensive techniques grounded in labels. u319–322 · 1:40:55 The lecture distinguishes improving a faster model through distillation from training the strongest model in a family.
Effective LLM post-training requires balancing reward optimization, stable policy updates, and informative credit assignment, with PPO, GRPO, and on-policy distillation offering different solutions and limitations. How can reinforcement learning and on-policy distillation improve LLM alignment and reasoning while managing training cost, policy stability, and token-level credit assignment?

How it argues

thesis Effective post-training balances rewards, stability, and token-level credit How can reinforcement learning and on-policy distillation improve LLM alignment and reasoning while managing training cost, policy stability, and token-level credit assignment?
1. theme Rewards define what reinforcement learning improves Reinforcement learning optimizes expected reward, using learned preferences or verifiable outcomes to guide model behavior.
  1. idea Optimize expected reward for preferences and reasoning
    1. mechanism Policies choose tokens or complete responses
    2. mechanism Pairwise preference modeling supplies subjective rewards
      1. implication Learned rewards replace repeated human judgments
    3. mechanism Verifiable rewards score checkable outcomes
2. theme Policy gradients need practical safeguards Reward-weighted updates enable learning and sample reuse but introduce variance, credit-assignment problems, and risks from unconstrained optimization.
3. theme PPO stabilizes learning at a cost PPO combines value-based advantages with constrained policy updates, but maintaining a value model adds substantial training burden.
  1. idea PPO improves credit and stability with costly values
    1. mechanism Values estimate future rewards from partial completions
      1. implication Value learning must track the changing policy
    2. counterargument Token advantages extend beyond immediate value differences
    3. mechanism Current-to-old ratios correct for reused batch samples
    4. mechanism Clipping stops incentives beyond the update threshold
  2. idea Fixed reference and batch-generating old policies differ
4. theme GRPO trades value models for comparisons Group-relative rewards remove the learned value function while leaving normalization biases and coarse outcome-level credit assignment.
  1. idea GRPO replaces learned values with group-relative rewards
    1. mechanism Verifiable outcomes can replace learned reward models
    2. counterargument Response-length normalization can encourage longer bad answers
      1. action Change normalization to equalize token contributions
    3. action Adjust reward normalization and asymmetric clipping
    4. counterargument Outcome advantages obscure token quality; process supervision costs
5. theme Distillation supplies contextual token-level feedback On-policy distillation supervises the student's own trajectories token by token, but its benefits depend on teacher quality and the distillation objective.
  1. idea Teachers supervise tokens on student-generated trajectories
    1. example Earlier student mistakes shape later teacher judgments
    2. evidence Toy ablations narrow performance gaps left after SFT
    3. counterargument Teacher availability, quality, and coverage limit distillation
    4. counterargument Leading models may benefit less than faster students
  2. idea Reverse KL asymmetry encourages mode-seeking students
    1. counterargument Reverse KL alone is not established as superior
6. theme Alternative teachers extend distillation's reach Answer-conditioned self-distillation and domain-specific expert teachers offer ways to preserve knowledge and overcome teacher limitations.
  1. idea Answer-conditioned models can teach their own students
    1. example GLM 5 uses self-distillation to preserve knowledge
  2. action Distill from domain experts trained with verifiable rewards
    1. argument Multiple teachers alleviate forgetting and competing objectives

The source, passage by passage

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

u1 0:00 Introduction
u2 0:05 Hello, everyone, and welcome to lecture 4 of CME 285. So today is a very special day because we'll be talking about reinforcement learning-based training methods that are actually getting quite popular, especially when it comes to post-training LLMs.
u3 0:26 And the reason why today's session is a little bit special is because contrary to the previous ones, there is going to be a little bit of math involved, which is why last time we said all the RL-based methods we will just reserve them for today.
u4 0:44 So today is the day. And yeah, so I wanted to start with this disclaimer. And the second thing I wanted to say is we will try to start from intuition first and then stay rigorous on the math. But please don't be intimidated by the big formulas.
u5 1:03 Yeah, hope it's going to be a good one. With that, we're going to start as usual by recapping what we saw last time. So if you remember, last time we were talking about training LLMs. And the first step of training LLMs is called pretraining.
1:10 · Recap: pre-training as learning to autocomplete · A Stanford ICME lecture slide recapping the previous episode. It presents pre-training as learning to autocomplete, illustrated with a Wikipedia "Teddy bear" article and a GitHub code file as training text, and the cross-entropy loss formula. A small inset shows the lecturer standing at the top right.
u6 1:24 It is a step that is meant to teach the model how to understand texts, how to understand codes. And this step typically involves training your model on a lot of data. By a lot of data, I mean trillions of tokens. So there's a lot of work around finding the right mixtures, how
u7 1:47 much web should it be in there, how the books should be in there, and so on. And the loss function for this one is the cross-entropy loss, which tries to penalize any deviation from the token that the model needs to predict, so its next token prediction. And we saw some emerging stages, such as mid training which
u8 2:12 involves higher quality data and also long context training, which involves training your model on longer contexts data as well. So that's the first step. The second step that we saw was called Supervised Fine Tuning, a.k.a.
2:23 · Recap: supervised finetuning teaches models to follow behavior · A Stanford ICME lecture slide recapping the previous episode. Pre-training ("learn how to autocomplete") is greyed out, while supervised finetuning ("learn how to follow a certain behavior") is highlighted, with two teddy-bear prompt/response examples and the SFT loss formula. A speaker is visible in a small inset at top right.
u9 2:29 SFT. And in this step, we're trying to take our model that understands text, that understands code, and tune it to be useful. So what do I mean by useful? For instance, if you want to have an assistant, you would want your model to give answers to questions.
u10 2:50 And that is one of the examples where people would use SFT for. So in terms of the loss function, we still have the cross entropy. Except here we're only trying to penalize deviations for token predictions that are on the output side.
u11 3:09 So it's next token prediction but for outputs. At the end of this stage, our model understand text. Our model is useful. But here we may want to inject human preferences. And we saw this third step called preference tuning. And this is the step that involves RL-based methods
3:14 · Recap slide: preference tuning stage of LLM training · A Stanford ICME lecture slide titled "Recap of last episode..." shows three training stages as boxes. Pre-training and supervised finetuning are faded, while "Preference tuning" (thumbs-up icon, "Learn how to align with human preferences") is highlighted. A loss symbol "L = ?" below poses the question of what the loss is for this stage. A small inset shows the lecturer standing at the top right.
u12 3:33 that we're going to see today. We did not see any specifics about the exact loss formula. So we're going to see today. And the other piece that we saw was the fact of increasing the model's reasoning capabilities, so
u13 3:53 reasoning, which also happens to be performed with this RL-based techniques. And we're also going to see that today. And this is where we left off last time. And today, we're going to focus on these last two stages, and in particular, emphasize how we come up with the loss, which
4:06 · Recap slide: preference tuning and reasoning are today's focus · A Stanford ICME lecture slide titled "Recap of last episode…" shows four stages of training a language model in boxes. The first two, pre-training and supervised finetuning, are faded out. The last two, preference tuning and reasoning, are highlighted with a red border and labeled "Today's focus". A small inset video at top right shows a person standing in a lecture room.
u14 4:17 right now, we don't know what the loss is and see how we can use that. Cool. I want to, again, have a big disclaimer here that these methods, they are typically mathematically involved, not just for this lecture,
u15 4:34 but if you were to read papers on that topic. And so if you feel like it's too much, here is the proposed steps for remediation. So first one, focus on the intuition on why a certain thing is needed.
u16 4:53 Second one is if there are formulas or symbols that you are not familiar with, just make sure to review them. And then the third one is just to interpret the formulas from first principles. And I actually have a second disclaimer, which is that on top of all of that, papers,
5:08 · Important note on conventions: "r" as reward or ratio · A Stanford ICME lecture slide warns that notation in the field varies. The symbol "r" could mean reward or the policy ratio ρ = π_θ / π_old. A small inset in the top right shows the lecturer standing in a classroom.
u17 5:14 they tend to choose conventions that are different from one paper to another. So that may add to the confusion. So we will see that, for instance, we're going to talk about rewards. Rewards are usually denoted r. But sometimes in papers, r could also
u18 5:32 mean a ratio, a ratio of probabilities. So here, to make sure we distinguish them properly, we will note them rho in this class. Second one is the policy, and we will see what a policy is. There are several kinds of policy.
u19 5:52 Is it the old policies, the reference policy? And sometimes these notations are also mixed. And then the third one to call out is that so far, we've been focusing on minimizing a loss function. And we noted the loss function L. Well, here
u20 6:13 most of the time we will want to maximize quantities. But the problem is sometimes these quantities are also noted L which usually you think of as being a loss. But just make sure that you know what you want to do. Do you want to minimize? You want to maximize?
u21 6:30 So that's also another thing to keep in mind. Cool.
u22 6:35 RL setup
u23 6:35 Well, with all of that said, let's start. And I wanted to first start with a refresher on what reinforcement learning is. So I'm not sure if any of you has ever taken a reinforcement learning-based class. But at a very high level, the way it works is you have an agent that is interacting
6:35 · Lecture slide on notation conventions in reinforcement learning · A lecture hall at Stanford shows a presenter standing in front of chalkboards, looking at a projected slide titled "Important note on conventions". The slide warns that the field uses many different notations. Its first two rows are faded, and the last row is highlighted: L(θ) could be a loss to minimize or an objective to maximize, J(θ).
u24 6:57 with an environment that is in a given state and that wonders what action to take to go to another state. So the way it tries to decide on which action to take is something that is leveraging a policy.
u25 7:18 So pi theta is typically how it's denoted. And after the agent performs an action, it receives a reward. And the goal of RL-based problems is to maximize the sum of rewards
u26 7:36 that your agent is collecting. Now you may be wondering, well, how does that have to do with LLMs? So here, we're going to see a little bit how things map to the LLM world. So the agent here is the LLM that takes some input as a state and tries to decide what to do next,
7:46 · RL formulation for LLMs (lecture slide, LLM highlighted) · A Stanford ICME lecture slide shows the reinforcement learning loop applied to language models. Only the "LLM" box is shown in full; the Environment box, the reward, state, action and policy labels are faded, which suggests they appear in later animation steps. A small inset at top right shows the lecturer standing beside a blackboard.
u27 8:02 so what token to sample next. So the action is a token. And in order to decide what token to go next to, what it does is it relies on the probability distribution that it outputs, given an input.
u28 8:20 And so here that one will be noted pi theta. So back to the notation last time, so this is where the pi is coming from, pi as in policy. And it collects rewards. So one note here is that it can be challenging to define token-level rewards.
u29 8:43 So a lot of the time, we will also encounter this setup not at the token level, but at the response level or at the completion level. And so here you can also consider an input as being your prompt, an action as being your response, and pi theta
u30 9:06 as being just the probability of you outputting a response, given your prompt. And then as a result of that, you will get a reward. So we're going to spend a few minutes on the reward part. And I know we spent maybe 10 minutes last time talking about this in different settings.
u31 9:29 So here I just want to summarize what we saw.
9:32 · Stanford ICME lecture slide: Transformers & LLMs, reinforcement learning outline · A split title slide from a Stanford University ICME course on Transformers & Large Language Models. The left side has the Stanford logo and course title; the right, on a dark red background, lists reinforcement learning topics with "Reward design" highlighted as the current section. A small video inset at top right shows a person standing in a lecture room.
u32 9:32 Reward design
u33 9:33 So in order for you to quantify how good a response is, you actually have typically two different categories of situations you can be in. The first one, the first category is where there is no absolute ground
u34 9:52 truth to the answer, where actually judging your answer is a little bit more subjective. And this is where we saw in the preference tuning case where we could have human preferences that try to up rank, let's say, responses that have, let's say,
u35 10:13 a better tone compared to ones that have a worse tone. And this is something that you would typically model. You would model with a reward model that leverages what we saw last time, the Bradley-Terry formulation, which is a way to express your loss in a pairwise fashion
u36 10:38 and train a reward model. So here the first category is you don't have a ground truth. And you try to come up with a way to quantify how good responses are, for instance, with human preferences. And you would use the Bradley-Terry formulation, and these methods would be called RLHF,
u37 11:02 because you would use human feedback as a way to quantify how good the responses are. And then the second big category of situations you can be into is where you can actually verifiably tell if your answer is correct or not.
11:10 · Types of reward: subjective (RLHF) versus verifiable (RLVR) · A Stanford ICME lecture slide compares two kinds of reward in reinforcement learning. The "Subjective" column (human preference, RLHF) is faded out, while the "Verifiable" column is highlighted. It is described as the solution of a problem, deterministic, with reasoning as the example (RLVR). A small inset shows the lecturer standing at the upper right.
u38 11:24 And so here are some examples of situations could be if you want to solve a math problem, you know what the solution is. Or if you want to write a piece of software, you know if your solution is correct or not if, let's say, it passes the test cases. So you have a deterministic way to check whether your answer is
u39 11:48 correct or not. And so these methods you will also find them under the name RLVR, Reinforcement Learning with Verifiable Rewards. And these are typically in the field of reasoning, so math problems and coding problems.
u40 12:09 So this is how you can have your rewards. And so it would be more at the completion level, not at the token level, at the completion level. So given this definition of reward,
12:23 · Motivation: preference tuning of an LLM via reinforcement learning · A Stanford ICME lecture slide titled "Motivation" states the goal: change model weights to increase rewards. A flow diagram shows a prompt going into a trained LLM, whose response is scored by a frozen reward model (RM) that gives a thumbs-down, and that signal feeds back to the LLM via RL. A small inset shows a speaker standing at the right.
u41 12:23 Policy gradient
u42 12:24 you may want to tweak the parameters of your model to maximize the rewards that you're getting. So coming back to the examples that we saw last week in the preference tuning case, what you want is to align your model with human preferences. So you would start with a prompt,
u43 12:44 and then you would generate a completion with your LLM. And then given the completion that you would generate, you would pass in the prompt and the completion into a reward model that is frozen that tells you whether the answer is good or not with respect to the prompt.
u44 13:05 And depending on how the reward model responds, you would tweak the parameters of your LLM in order to maximize the rewards, the expected rewards that is generated. So this is the preference tuning case
u45 13:22 and you have similarly for the reasoning case, when you have a prompt that is maybe more of a reasoning prompt or a coding prompt. And here you have a deterministic way to evaluate the quality of your answer. And so here you would only look at the final answer.
u46 13:44 And you would use that in order to tweak the parameters of your LLM which is here, your policy. OK. So now we talked about the setting. Now let's go into the math. Yeah?
u47 14:00 [INAUDIBLE] So the question is, would the probability of a full response be very low? Yes, indeed it would. That is the reason why people use logarithm in order to have some numerical stability. But yeah, great point. Yeah? [INAUDIBLE] So the question is, do you need a human every time you generate a completion?
u48 14:20 So the answer is no, which is the reason why you're actually training a reward model firsthand. You're training it at human preferences. And the reason why you're doing that is because you don't want to have a human in the loop every time you generate a completion.
u49 14:37 And what you would do is to use that reward model for any responses that you're generating, and you're using that reward model in order to assess how good the response is. So yeah, you are actually not leveraging humans in this loop. OK.
u50 14:55 Perfect. So I was saying let's go into the math. So here, our objective function is to maximize the expected rewards that you're getting. So in mathematical terms, what you want to do is to find parameters theta of your model
14:58 · Motivation slide: maximizing expected reward over model weights · A Stanford ICME lecture slide titled "Motivation" states the goal: change model weights to have more generations with higher rewards. It gives the objective as maximizing J(θ), the expected reward R(τ) over trajectories τ sampled from policy π_θ. A small inset in the top right shows a person standing in a lecture room.
u51 15:18 that maximize the objective function, which is the expectation of rewards as you're sampling trajectories from your policy pi theta, which is your model. And so I know what you're thinking.
u52 15:41 In machine learning, when you want to optimize an objective function, typically what you want to get is the gradient because you want to perform your updates. So the first question that we want to tackle here is how do you get the gradients of this objective? And so I know a lot of us are coming
u53 16:00 from different backgrounds. And I want to make sure we're speaking the same language. So little refresher on gradients. You can think of gradients as a way for you to get the direction and the magnitude of the steepest increase.
u54 16:22 So in the 1D case where your function has an input x, that is just one number, you all know the derivative. So if the derivative is positive, what that means is your function increases as x increases. If it's negative, it increases as x decreases.
u55 16:45 So the generalization of that for the n dimensional case is the gradient. And the gradient of f in this case is the partial derivative of that function with respect to all the dimensions of your input x, which is in Rn. Are we all good on this?
u56 17:10 OK. So with that being said, let's go back
17:12 · Policy gradients: objective J(θ) and unknown gradient · A Stanford ICME lecture slide titled "Policy gradients" defines the objective J(θ) as the expected return R(τ) over trajectories τ sampled from policy πθ. An arrow leads to the gradient ∇θJ(θ) = ?, posing the question of what it equals. A note says the derivation uses properties of the log function and rearranging terms. A small inset at top right shows the lecturer standing in a classroom.
u57 17:12 Derivation of the gradient
u58 17:14 to our original problem, which is we have an objective function to maximize. And what we want to do is to compute the gradient. And so I will just do that. So what we want is for us to maximize the expected rewards. And so in mathematical terms, what that means is
u59 17:42 we want to maximize the rewards such that the trajectory is sampled from your policy. So this is exactly what is on the slide means. And so if you want to take the gradient of that,
u60 18:05 well, you would just compute what is in between the integral. So what you would do is just wander for a second what is a function of theta. So is R a function of theta?
u61 18:23 No, it is frozen. Is pi theta a function of theta? Yes. So what you would do is actually go and put the gradient in there. And of course, you have some regularity that you assume is in place.
u62 18:44 OK. Mathematical trick. What is the gradient of pi of theta? So you may know that the gradient of log of a function is the gradient of that function over that function. So that's a mathematical result.
u63 19:03 So what that means is that the gradient of a function is equal to the gradient of the log of that function times that function. So here this is equal to the gradient of the log of pi theta times pi theta.
u64 19:28 And so what does that mean? I'm going to just rewrite everything so that we're all clear. So it's integral of R tau gradient of log of pi theta pi theta of t and d tau.
u65 19:56 So this is nothing else. This is nothing else than the expected value of your rewards times the gradient of the logarithm of pi theta,
20:13 · Lecturer at chalkboard deriving policy gradient equation · A person in a green shirt and glasses points at a chalkboard in a lecture hall. The boards show a policy gradient derivation: an integral of R(τ) times the gradient of log π_θ(τ) times π_θ(τ) dτ, then being rewritten as an expectation E[R(τ)...]. A "Stanford" watermark appears at the lower right.
u66 20:25 where tau is sampled from your policy of parameters theta. Does everyone agree with the math? Yes? [INAUDIBLE] So the question is, how did you introduce the log? So I just used the fact that the gradient
u67 20:48 of the log of a function phi of theta is equal to the gradient of that function over that function. So it's just a mathematical identity. [INAUDIBLE] Yes, exactly.
u68 21:07 So what we did is we just put that here. [INAUDIBLE] Yes, exactly. Thank you. Great. Does everyone agree with the math? OK, cool.
u69 21:25 So the reason why I'm doing the math on the blackboard is not because I like writing on the blackboard, but I mean, I do. But here you will see this formula quite a lot. And it's just important to know where this comes from. And so what that means is that the gradient of your objective function is incentivizing your model
u70 21:49 to increase probabilities of actions that have a positive reward and decrease the probability of actions that have a negative reward. Yeah? [INAUDIBLE] So the question is, what does the tilde in the tau tilde
u71 22:10 pi of theta mean? So that means that you're sampling your trajectory from the probability distribution given by your policy of parameters theta. So tilde means sampled from. So again, I will just repeat the intuition.
u72 22:32 So what you can take away from this formula is that if the reward is positive, the gradient will incentivize your model to increase the probability of such actions. And it will incentivize your model
u73 22:52 to decrease the probability of actions that have negative reward. OK. So based on that, there's one little thing that I want to tell you about, which is that we talked in terms of trajectories. And in the LLM case, what we typically do is we first have
23:01 · Trajectories in RL versus LLM expectation notation · A Stanford ICME lecture slide compares the general reinforcement-learning expectation over trajectories with the LLM case. In the general case the expectation is over a trajectory τ drawn from policy π_θ. For LLMs it becomes a nested expectation: a prompt x drawn from a prompt set D, then a response y drawn from the LLM's response distribution π_θ(.|x). A small inset shows the lecturer standing beside a board.
u74 23:15 a set of prompts, the x's. And then we sample the responses from the policy of our model, given these prompts. So the reason why we only did the tau sample from pi theta is just because it didn't want to make it more
u75 23:39 confusing on the blackboard. But just think of expectations as being over prompts and then over responses, given these prompts.
u76 23:50 REINFORCE
u77 23:50 OK. So now you may wonder, well, how can you update the weights of your model given what we saw? So here what we would do is we would have a prompt x, and we would generate a completion y sampled from our LLM, given the input x.
23:50 · Lecture on policy gradients for LLMs at Stanford · A man in a green T-shirt stands in a lecture hall facing a projected slide titled "Trajectories in our case". The slide compares the general reinforcement-learning expectation over trajectories with the LLM version, which takes an expectation over prompts and then over responses from the model. The chalkboards behind it hold policy-gradient derivations.
u78 24:14 What we would do is we would evaluate the reward that is associated with the prompt and the response. And then we would compute the gradient, which we saw is R times the gradient of the log of pi. And so here's exactly what's on the slides.
u79 24:35 And then you would update your parameters. So you may be familiar with gradient descent. So it's theta minus, but here we want to maximize the objective. We want to maximize our model generating high reward. So here it's actually theta plus alpha times the gradient
u80 24:56 because it's gradient ascent. So one thing that I want to say is that contrary to pretraining, contrary to SFT, here we need to always generate the response. Because here we're in an on policy setting.
u81 25:16 We're generating the response and then performing the update on the response generated by the model. This is not a fixed response. It is generated by the model. So the problem with that is you're asking your model to generate responses every time you want to make an update. And that step is very expensive.
u82 25:38 You cannot parallelize it because you need to ask your model, OK, what is the next token? And then you take that token and you ask, what is the next token? And one at a time you're generating these responses. And so one thing that I want to tell you
u83 25:51 Importance sampling ratio
u84 25:53 is a method that people can use in order to limit the number of times they need to generate completions. And I want to write that down.
u85 26:14 So here what we're saying is that we want to maximize the expected rewards when the trajectory is sampled from the current policy. So we want to maximize this. So we want to maximize the expectation of R of tau
u86 26:38 when R, sorry, when tau is sampled from the current policy. Which is equal to the integral of R of tau pi theta of tau d tau. But another way of writing this could be to say if we have,
u87 27:05 let's say, another policy tau fixed, we could express this quantity as a function of tau fixed by letting this be popped up.
u88 27:23 So here we do pi theta of tau over pi theta fixed of tau times pi theta fixed of tau d tau. Which is equal to the expectation
u89 27:47 of R of tau times pi theta of tau over pi theta fixed of tau. Where tau is sampled from pi theta fixed.
27:51 · Lecturer at chalkboard deriving policy gradient importance-sampling objective · A still from a Stanford lecture video shows a person with their back to the camera, standing in front of a chalkboard. The board shows the reinforcement-learning objective of maximizing expected return E[R(τ)] with τ drawn from π_θ, written as an integral. Below it, the integral is rewritten by multiplying and dividing by a fixed-parameter policy, which is an importance-sampling step.
u90 28:15 So I want to just highlight. So we went from maximizing the expected value when tau is sampled from pi theta of a quantity over to a quantity which samples the trajectory from a policy
u91 28:37 of fixed given parameters. So why did I do that? So let's imagine for a second that we're tired of always generating completions every time we want to perform an update. Let's assume we generate completions, let's say once.
28:44 · Lecture on REINFORCE algorithm at Stanford ICME · A lecture room still shows a man in a green shirt writing on a chalkboard covered with policy-gradient derivations. A projected slide on the right titled "(Vanilla) REINFORCE, a Monte-Carlo algorithm" lists four steps, with Step 1 (sampling a response) highlighted and marked as expensive.
u92 29:00 And then we perform updates using the completions of that one time but by reweighting the rewards for a given trajectory with the ratio of the probability of your current policy
u93 29:21 over your fixed policy. So the reason why I wrote that is because sometimes we don't want to always do that. We want to have a fixed policy that does that once and then just reweight the terms in a way that makes it still mathematically valid,
u94 29:41 because this is what we care about. So one thing that I want to add is this term is called importance sampling ratio. Yes?
u95 29:59 [INAUDIBLE] So the question is I don't get the benefits. We're still computing pi theta of tau. What is the benefit? So the trick is evaluating the policy is different with generating a rollout. Because evaluating the policy, you're
u96 30:17 taking everything as input in parallel and computing everything, all the probabilities. But here when you do the rollout, as in when you generate a response, you need to take the first token to generate the token. And then take that token and generate the next token over and over.
u97 30:37 So this is expensive. This is cheap. Yeah? [INAUDIBLE] So the question is, please explain a bit more what you mean by this is cheap. So here you know the full response. You know the full response. So you're putting all the responses inputs, all in parallel.
u98 30:58 So it's all being computed in a nice parallel fashion. So maybe I can do something else that could be more helpful. You mean that [INAUDIBLE]? Yeah, exactly.
u99 31:22 So here, so generation, what you're doing is you're putting-- so this is your model-- putting a token in, token out. That token in, token out, and so on.
u100 31:53 But for evaluation, what you're doing is you already know the full completion. You already know it. You know all the tokens. So you're putting everything as inputs all at once.
u101 32:15 And you know that your LLM is a decoder only LLM, so it has the masked self-attention. So it's able to nicely only attend to itself. And the ones before all that is very parallel. And you can get the corresponding probabilities as outputs.
u102 32:34 This is expensive. This is cheap. OK. Great question. Yeah? [INAUDIBLE] So the question is we're still doing the expensive calculation with theta fixed. Yes, but you can do it once and reuse that several times. You don't need to do that every time.
u103 32:53 [INAUDIBLE] The question is for multiple training steps. Yes. So we will see that more in detail. I just didn't want this to come out of the blue. This is the reason. So now, what I want to do is to go through some challenges that such a vanilla approach can exhibit.
u104 33:15 So this is not one challenge. This is just like a small FYI.
u105 33:22 Challenges of policy gradients
u106 33:23 The first challenge that I want to mention is that if we use a reward function that does not have a concept of relative compared to some baseline, we may get gradients that are of high variance.
u107 33:45 So I'll give you an example. Let's assume we have a prompt, and we have two answers. Let's say one is good and the other one is bad. In the first case, the good answer has a reward of 101, and the bad answer has a reward of 100.
u108 34:05 So you know that 101 is greater than 100, but both are very high, and you don't exactly get the difference relative to one another. The problem is that this will lead-- this will lead to noisy gradients,
u109 34:23 because you know that your model will update parameters as a function of R and the gradient of the log of the policy. So if R is always big to make big updates, it will make big updates to get good answers pop up more.
u110 34:48 It will make big updates to make the bad answer pop up more but in a lower magnitude, because the reward is lower than the one for than for the good one. So at the end of the day, if you do that enough times, you will get the parameters that you want to get, but you will have very high variance gradients.
u111 35:09 As in you will have big updates for good, then big updates for bad and so on. So what you want is to actually get a sense of what is the expected reward that you should receive for a given state.
35:16 · Challenge 1: High variance gradients, missing reward baseline · A Stanford ICME lecture slide shows a bar chart of reward for a good answer and a bad answer around a dashed baseline at 100.5. The good answer sits +0.5 above the baseline (thumbs up) and the bad answer 0.5 below it (thumbs down). It illustrates that without a relative baseline, rewards are not compared against a reference, which leads to high variance gradients. A small inset shows the lecturer standing in front of a chalkboard.
u112 35:28 And then contrast the reward of each action with respect to that. So if you say that OK, on average 100.5 is roughly OK, then if you have plus 0.5, then you know it's good. And if you have minus 0.5, then you know it's not good.
u113 35:54 And you can show on a toy example that the gradients that you get as a result of that are of much lower variance when you do have that baseline, when you do have that sense of relative to something.
u114 36:13 So that's the first challenge. The second challenge is that the two types of reward that we saw are at the completion level, meaning that you either know that everything is good or either know that everything is bad. You either know that everything is correct,
36:14 · Challenge 2: No explicit token-level credit · A Stanford ICME lecture slide argues that a correct final answer doesn't mean every token in the output was good. It gives a prompt about Teddy in a sealed box moving to the kitchen. The output reaches the right answer, "Kitchen", through a flawed step in which Teddy climbs out of the box. A small inset shows the lecturer standing near a blackboard.
u115 36:31 or you know that everything is not correct. Well, I'll tell you this example. So let's assume your prompt is about Teddy who is in a sealed box. And you want to-- you're actually moving the box to the kitchen. And now you're asking yourself where is Teddy? And let's assume your model says, OK, step 1,
u116 36:50 Teddy climbs out of the box. Step 2, teddy walks to the kitchen. And step 3, answer-- kitchen. So know that the answer is correct because you move the box to the kitchen. But what is not correct is Teddy who's working because Teddy is in the box the whole time.
u117 37:12 And in the vanilla case, you would incentivize all tokens of this response to happen more frequently. And this is not what you want to do. So you're lacking an explicit token level credit assignment. That is something that we need to work on.
37:25 · Lecture slide on missing token-level credit in RL · A lecturer stands in front of chalkboards in a classroom while a projected Stanford ICME slide titled "Challenge 2: No explicit token-level credit" is displayed. The slide uses a "Teddy in a sealed box" prompt to show that a correct final answer can follow flawed reasoning steps, so not every token deserves equal credit. Chalkboards show a policy-gradient derivation and sketches labeled Generation and Evaluation.
u118 37:34 And then the third one I think we briefly mentioned it last time. The policy that you already have at the beginning of this training stage is already at a good state. You already did pretraining. You already did your SFT. Your model is useful, and you don't want your trained model to deviate too much from the model
u119 37:58 that you trained because it's already good. And on top of that, sometimes the rewards that you have, they're not exactly aligned to your end goal. So I'll give you the following example. So let's assume me I come here, and in order to know if I did a good job at giving this lecture,
u120 38:24 I would rely on the intensity of the claps that you would make at the end of the lecture. So if I were to only optimize for that, then that would not be ideal, because maybe I would just spend an hour or two hours making jokes that are not related to the class.
u121 38:42 I would still receive claps, maybe, but it would not be aligned with my objective. And so there is this risk of reward hacking, where your reward model, may actually be imperfect. And you may have your model who's
u122 38:59 just trying to game the reward model, and you don't want that. So I hope I convinced you that we don't want the policy. We don't want the model to be too far from the model that you already have. So we want to have some sense of constraints in there.
u123 39:21 And we will see how we can quantify the difference between distribution. Yeah? [INAUDIBLE] So the question is, what do we mean by don't want to regress on previous tasks? So your model was trained for specific things. And you're now asking it to be better at something else. And so the capabilities that it has acquired
u124 39:42 can be something that it loses if you emphasize too much on the task of interest here. And so you also want your model to not deviate too much because of that. OK.
u125 40:00 Cool. Summary. What are the three challenges? The first one is gradients can be noisy. Second one is we don't have out-of-the-box an explicit token-level credit assignment. And the third one is we don't have any explicit constraints on how far the policy can go with respect to the policy
u126 40:24 that we're starting from. So with these three challenges in mind, what we will do is talk about the most common RL algorithms
u127 40:34 Proximal Policy Optimization (PPO)
u128 40:36 that are out there, which I'm sure you've heard of. We will start with the first one, PPO. So PPO stands for Proximal Policy Optimization. And what it tries to do is it tries
u129 40:54 to maximize your objective function but with two differences compared to the first one. The first difference is it tries to maximize advantages and not rewards. And the second difference is it tries
u130 41:12 to not deviate too much from the policy that you had before. And we will see exactly how these two terms are obtained. But just one note. PPO, the P in PPO is for that second term. You want a bit of proximity to that policy of reference.
41:22 · Proposed solution: PPO (Proximal Policy Optimization) lecture slide · A Stanford lecture slide contrasts REINFORCE, which maximizes rewards, with PPO, which maximizes advantages and adds a term for not deviating too much from before. An arrow from the highlighted "P" in "Proximal" points to that "Don't deviate too much from before" box, explaining what "proximal" means. A small inset shows the lecturer standing by a blackboard.
u131 41:35 So with that being said, let's go into the first term, which is the advantage term. So the reward function that you have is wondering how good is this answer with respect to the prompt. But what we really want to ask is,
u132 41:55 compared to what we would expect for this prompt, how good is this response? So we want to have some form of relative to a baseline, and that is what advantages are aiming for. And in order to illustrate that, let's go through an example.
u133 42:17 So let's assume that we have the following prompts. Teddy is in a box. Where it's Teddy? And then you have your response-- in the box. So the reward for that one-- let's assume it's a correct response. So the reward for the whole completion relative to that prompt is, let's say, plus 1.
u134 42:37 But you want to know how these different tokens are actually receiving the credits relative to what they are expecting.
42:49 · PPO computes advantages, not just rewards (lecture slide) · A Stanford lecture slide on Proximal Policy Optimization. It shows an LLM taking the input "Teddy is in a box. Where is Teddy?" and producing the output "In the box.", with a final reward of +1. The slide says PPO maximizes advantages, based on the expected final reward given the prefix so far, and adds a faded term to not deviate too much from before. A small inset shows the lecturer standing by a blackboard.
u135 42:49 Value function and advantage
u136 42:50 So the question that we want to ask ourselves is, given a partial completion, what is the expected final reward if you were to continue sampling from the policy for the rest of the tokens?
u137 43:10 So I'll give you an example. So let's assume you have the whole input. And then you're decoding, let's say, in here you want to ask yourself, OK, if I were to continue like this on average, what would be the expected final rewards? And here, let's suppose it's 0.6.
u138 43:31 Then you decode. You go to the, OK, 0.7 box. It's the correct answer. So it should have a big bump, so let's say 0.95 and 1. So what did we do here, what we tried to do is to ask ourselves, given a partial completion, what the expected final reward would
u139 43:56 be. And the nice thing with this approach is we have a sense of a baseline. And the baseline here is compared to how we would have completed if you were to sample the policy.
u140 44:15 And you can exactly pinpoint the contribution of a given token when you do that. So in this example, if you were to compute the very simple version of the advantage for the token box, you would take the value with box
u141 44:40 and subtract the value without box. So it's 0.95 minus 7, so plus 25. So I hope this example motivates the fact that we're introducing a new function, which here we call the value function.
u142 45:03 That is a function that is defined at the token level. So what that function does, so the math formula on the slide, what it says is it tries to determine what is the expected future rewards if you were to start
u143 45:22 from the input and the partial completion and if you were to continue decoding by sampling from your LLM, by sampling from the policy, what would be the expected value of that? That is your baseline.
u144 45:40 So in this case, the model is actually trained along with the policy because you need to ask yourself what would be the expected reward if you were to sample from the policy? So you need to leverage the parameters of the policy that you're training.
u145 46:02 Here, the way you would train such a model is you would try to predict the reward at the end of the completion, given a partial completion along with the prompt. And this quantity would be defined at the token level. So for each token of the output, you would have a value.
u146 46:25 So this is the value function. And the way you would compute the advantage is by taking the value up to that token and do minus the value up to the token right before. And here you're able to tease out
u147 46:47 the impact of that token itself. So one thing that I want to say it is actually more complicated than that. So this, that's why it's in quotes. So it's a quote unquote. The reason why it's a little bit more complicated
u148 47:05 than that is if you were to generate a token, you may increase the likelihood of you getting a high reward, let's say. But you may also put yourself in a position where future tokens would benefit from the token that you generated.
u149 47:26 So let me give you an example. Let's assume you're teaching a concept to someone. You're teaching it some math concepts. You teaching the concepts may lead to your friends acing the math exam.
u150 47:45 So that's a short-term reward. But you can also assume that the fact of them succeeding the exam is also placing them on a good path for succeeding later exams and maybe being successful at work
u151 48:04 and then starting a company, being a CEO, being a billionaire. So you cannot completely attribute you teaching the concept to your friend as being the sole reason for them being successful in the future. But there is some part of it that should be attributed to that. And that is the idea behind the generalized advantage estimation
u152 48:27 paper, which is not part of the exam. It's not part of the scope of this class. But in case you're interested, feel free to look at it. The idea is to also look at how future values also perform and link that back to a given token.
u153 48:50 Cool. So, so far, does that make sense? So here what we saw was a way to compute advantages, which are a quantity that tells you how a given token, predicted token is impacting your purpose of getting, let's say, high rewards or getting token level signal?
u154 49:13 And you also have a sense of a baseline by comparing your value with the value without that given token. Yes? [INAUDIBLE] So the question is, is there any interaction between the value function and the policy?
u155 49:30 So these two are actually two models. In practice, you train them jointly. So you would have one loss function and have a penalization for the value and the penalization for the actual policy. But in this case, these two are completely disjoint.
u156 49:50 The value function is not being fed as input anywhere. Yeah? [INAUDIBLE] So the question is the reward model and the value model are both trained. So the reward model is trained beforehand. And the value model is trained at the same time as when you're training your policy.
u157 50:10 And the main reason why you want to do that this way is because the definition of your value model is to determine the expected rewards if you were to continue according to the policies that you're training. Cool.
u158 50:32 Yeah? [INAUDIBLE] So the question is value function is a function of theta. So yes, there is a backbone that is shared. And the gradients are determined by your loss function, which contains a value loss function and then a policy, the actual loss function.
u159 50:52 So in case you're interested in the details, actually the PPO paper does have some technical details. They're interesting to look at. So now what we're saying is we're not trying to optimize the expected value of the rewards. We're trying to optimize the expected value
u160 51:12 of the advantages. Well, not exactly.
51:16 · PPO objective: maximize advantages instead of rewards · A Stanford ICME lecture slide explaining that PPO computes advantages, not just rewards. The objective J(θ) is split into a "Maximize advantages" term plus a faded "Don't deviate too much from before" term, and the expectation of rewards E_t[R_t] becomes the expectation of advantages E_t[A_t]. A small inset shows the lecturer standing in front of a blackboard.
u161 51:16 Policy ratio
u162 51:18 So if you remember this, so assuming you can all see the blackboard at the top, we that generating completions were expensive. So do we want to generate completions every time?
u163 51:37 No. So what you want to do is to leverage completions for multiple updates. And here you can do that by generating a batch using pi theta and then update pi theta on that same batch.
u164 52:00 And the pi theta would become pi old because it's not the current policy anymore. And so that is the reason why you would see in papers that ratio pop up. So you're not optimizing for expectation of the advantage. You're optimizing for the expectation of the ratio
52:07 · PPO computes advantages and uses a policy ratio · A Stanford ICME lecture slide explaining that PPO's objective maximizes advantages rather than raw rewards, with a second term (faded) that keeps the policy from deviating too much. Because the same batch is reused for multiple updates, the objective uses the ratio of the current policy to the old policy multiplied by the advantage. A small inset shows the lecturer standing at the right.
u165 52:19 times the advantage. And this is because you want to adjust for the fact that you're actually sampling from the old policy. But you want to make sure that this is corresponding to the current policy. So you have this corrective term, which is the ratio between the current and the old policy.
u166 52:43 And this is where I want to give a disclaimer. So some papers they note this ratio R. So this can be confusing with respect to the rewards. So here we note this rho, and it is called importance sampling ratio. You have a question?
u167 53:01 No? OK. Great.
u168 53:03 Ratio clipping
u169 53:04 So what can happen if you do things this way? Well, you can run into the issue that we discussed previously where you may run into a situation where you're updating your theta such that it deviates too much from previous policies.
u170 53:25 So what you actually want to do is to have some form of a control over that ratio. And you don't want that ratio to be, let's say, too far from one. So there is a formula for that. And I don't want you to look at the formula right now.
u171 53:44 I want us to just take a look at it from an intuitive perspective. So here is what I want us to do. So let's assume that we're interested in how much your model is updating the policy.
u172 54:05 So here we said it's something like the ratio times the advantage. So it's linear as a function of the advantage. So if you were to draw-- I'm going to just replicate what we have here--
u173 54:26 the evolution of this objective function, which I will note J, as a function of the ratio-- so if A is positive, so if your advantage is positive, meaning you want your model to increase the probability of a given action from occurring,
u174 54:46 then you will have something like this. Like, A is positive. Does everyone agree with me? So let's assume here we have 1. So I'm just going to recap.
u175 55:03 So rho is pi theta over pi old. So 1 means that your current policy and the old policy are saying the same thing. And so if you're here, then what you want to do in order to perform your update is to maximize J.
u176 55:27 So how do you maximize J? You maximize J by increasing the probability of your policy performing that action. So you would go more somewhere there. You would have your new policy be updated and correspond here. The problem is, if you don't do anything to that expression,
u177 55:52 the gradient is always nonzero and incentivizing you to increase rho to increase the probability of your policy generating that action more and more. So what you want to do is to break that incentive and clip that function in order for you to have a 0 gradient
u178 56:19 region where if you were to, let's say, perform an update to your policy and it would become here, you would have no incentive to increase rho. And this is exactly what is on the slides. What we want to do is to clip that function,
56:36 · PPO clipping prevents large policy updates (Stanford lecture slide) · A Stanford ICME lecture slide explains that PPO's objective J(θ) combines maximizing advantages with a term that keeps the policy from deviating too much from before. The first idea is to clip the ratio between new and old policy. The plot, for positive advantage (A > 0), shows J(θ) rising linearly with ρ(θ) up to 1+ε, then flattening. A red dot sits near ρ = 1 on the rising line.
u179 56:40 to remove the incentive from the update to increase the probability of an action being generated. And you want to avoid your policy from increasing that probability even more. So in other words, you don't want your policy
u180 56:59 to deviate too much from the old policy. So this is when the advantage is positive. Similarly, very similar fashion when it's negative. Here, the objective function will incentivize your model to decrease your ratio, to decrease the probability
u181 57:18 that your model has from generating an action that leads to negative advantage. So you will have something like this. So if you come back to the formula, this is what the formula is doing. Yes? [INAUDIBLE] So the question is you had introduced the KL divergence.
u182 57:40 Why don't you use that? Well, it's my next slide. So I will say a little bit about how that works.
u183 57:45 KL Penalty
u184 57:46 But this is one way of constraining your model from being updated too much. So to your point, let's go to the KL divergence. So what is the KL divergence doing? So it is an operator that is taking in two probability distributions, let's assume p and q.
u185 58:06 So here this operator is assuming that p is true, and it is weighting that by log of pi/q. You have also the continuous version, which is just take the integral.
u186 58:26 So you have a second way of quantifying deviations of your model by saying you actually want a penalization term in your loss function that compares the current policy and some other policy.
u187 58:45 So here typically what people do is they actually penalize your current policy from being too different from your SFT policy, from your base policy, your reference policy, which is why I want you to pay attention to the subscripts of every pi that you see out there.
u188 59:05 So here it's pi theta with respect to pi ref. Pi ref is reference, as in the base policy. And this contrasts with pi old, which is the policy that you use to generate the completions. So I'll give you a full story.
u189 59:24 You start with your pi ref. You train your model, so you generate a batch. This becomes pi old. You perform several updates, and then you generate another batch. This becomes your pi old, several updates. Pi ref stays fixed, and this is what you're comparing your current policy to.
u190 59:48 Connection with DPO
u191 59:48 And so I want to very quickly mention a scary formula we saw last week, which was the one from DPO. And I'm not sure if you remember, but we had said that there was a supervised way to express the RL objective by using
59:48 · Direct Preference Optimization loss function lecture slide · A Stanford lecture slide connecting to DPO (Direct Preference Optimization), showing the DPO loss rewritten in a supervised form. Two red boxes highlight the log-ratio terms for the winning and losing responses, and a red formula on the right defines the implicit reward r_θ(x,y) = β log(π_θ(y|x)/π_ref(y|x)). Faded bullets list the benefits of DPO. A small video inset shows a lecturer standing in a classroom.
u192 1:00:15 the Bradley-Terry formulation. And what we said was that the objective function was following something that was very much into the line of what the Bradley-Terry was saying, which is something like, so this one is like L minus expectation of log of sigma. And sigma is your sigmoidal which is sigma of, let's say,
u193 1:00:44 Rw minus Rl. And we had said that R was defined as beta log of your policy compared to some other policy. And here let's say it's a reference.
u194 1:01:05 And we had said that beta was a quantity, some quantity. So I just want to draw a connection here that the beta that you're seeing in the KL divergence, the beta that is actually penalizing for your model being too different from your reference
1:01:06 · Lecturer at chalkboard with preference-loss equation · A man in a green t-shirt and glasses stands in front of a chalkboard in a lecture hall, smiling and looking toward the board with one hand raised. The board shows a handwritten loss function, apparently the Direct Preference Optimization (DPO) style objective, with a reward expression written as beta times the log of a policy ratio. A faint "Stanford" watermark appears at the bottom right, and part of a Stanford-branded screen is visible at the top right.
u195 1:01:23 model, is actually the same beta. So the higher the beta, the more you will try to not deviate too much from your reference policy. And I just want to show you mathematically why that is the case intuitively.
u196 1:01:41 So sigma, if you were to draw the function, it's something like this. Whoop. Like this. It's like 1 and 0. So sigma of x is 1 over 1 plus exponential of minus x. So when x is 0, it's 0.5.
u197 1:02:08 If x goes to plus infinity, this goes to 1. And when x goes to minus infinity, this goes to 0. So there is a region where the sigmoid evolves very quickly. And so we had said that the incentive of your loss
u198 1:02:30 is for this difference to be as big as possible. And how do you do that? You want to do Rw as big as possible. And how do you do that? You do that by increasing the probability of your policy generating the winning answer.
u199 1:02:51 But if the better that you have is a big beta, then that means that even a slight deviation from the reference policy will make this whole value big. And you will already be in a region
u200 1:03:13 where you're already close to 1. So in other words, if you have a big beta, your policy will have a lower incentive to be far from the reference policy. And this contrasts with the lower beta.
u201 1:03:32 So if you have a low beta, then you need this quantity to be much higher than this quantity for this to be big. So I'm not sure if that helps. But all that to say, that beta is actually linked to your incentive to want to make
u202 1:03:56 this close to the reference policy or not. And this is the same beta that is in the DPO loss. Yes? [INAUDIBLE] So the question is rho t, is it using pi old? Typically, yes. And here it is pi ref. Yes? [INAUDIBLE] So just to confirm there are three policies.
u203 1:04:14 Yes, there are three policies. So pi theta is the current policy that you're updating. Pi old is the one that you use to generate the batch. And then pi ref is your base model. Yes, was it the same question? [INAUDIBLE] Yeah, so the question is, if you only care about how a policy deviates from its old value,
u204 1:04:34 how do you make sure that it doesn't deviate too much from the reference one? Typically what people do is they have these two constraints simultaneously. They have the clipping of the ratio with respect to the old policy, so the iteration constraints, along with the absolute respect
u205 1:04:54 to the reference policy one. So they're both penalizing for deviation with the old policy but also with respect to the absolute base policy. [INAUDIBLE] Yeah? [INAUDIBLE] The question is, how you compute KL?
u206 1:05:12 So it's a great question. Because KL is expensive because it's over the whole vocabulary, you have some estimator for that. Yeah. You had a question? No. OK. Great. Let's move on. So if I were to recap what we saw, we saw a way to maximize the quantity that is actually
1:05:24 · PPO nowadays: maximize advantages, limit deviation from before · A Stanford lecture slide (ICME) explaining that the PPO objective J(θ) has two parts: maximizing advantages and not deviating too much from before. It says advantages use a value function as baseline, and that staying close is done by clipping against the old policy or a KL penalty against a reference policy. A small inset shows the lecturer standing near a blackboard.
u207 1:05:33 as a function of, let's say, a baseline, which is called advantage. And here we used the value function to do that. And we also mentioned that we had some things in place in order for us to not deviate too much from what we had before. So we saw the clipping mechanism with respect to the old policy,
u208 1:05:55 but we also saw the KL penalty term between the current policy and the reference policy. So these are ideas. But some methods they mix both, some methods they use one but not the other. Some methods mix them. There are multiple flavors of it.
u209 1:06:13 But I think what is important for you to remember is the different methods that you can use in order to achieve these goals. And I wanted to have a dedicated slide to say if you were to read the PPO paper, please be careful. L is not a loss.
1:06:21 · PPO clipped objective formula, terminology side note · A Stanford ICME lecture slide titled "Side note: terminology alert" showing the PPO clipped surrogate objective L^CLIP(θ). The formula is a screenshot from Schulman et al., 2017, "Proximal Policy Optimization Algorithms". A small inset in the top right shows a person standing in front of a blackboard; the rest of the slide is blank.
u210 1:06:32 It is an objective function. You want to maximize that quantity. You don't want to minimize. And the second one is you will see R for the ratio. Be careful. It is not a reward. It is a ratio. And with that, I will just recap what we saw with respect to the challenges that we mentioned.
u211 1:06:52 Gradients being noisy, well, we have a way to compute a baseline, which also turns out to give token-level credit assignments. And we said we could also use either clipping or the KL divergence methods in order to ensure that the policy is not
u212 1:07:11 deviating too much from what we had. But the big problem with this approach is that you need to train a value function. That is a lot of hassle. And so this is the reason why this is not the only method
1:07:29 · Value function as fix for policy gradient limitations · A Stanford ICME lecture slide titled "...with a core limitation" lists three policy-gradient problems, faded out: noisy gradients, no token-level credit assignment, and no constraint on policy update magnitude. The highlighted box says to compute advantages with a value function, which addresses the first two. The slide notes the limitation that training a value function is a lot of hassle, marked with a frowning emoji. A small inset shows the lecturer standing in front of a blackboard.
u213 1:07:29 Group Relative Policy Optimization (GRPO)
u214 1:07:32 we're seeing in, this lecture. There's a second method, which has gained a lot of importance in the past couple of years, which is called GRPO. So GRPO stands for Group Relative Policy Optimization. And what it does is it tries to maximize advantages
u215 1:07:54 but without a value function. That is the big added value. But without the value function of GRPO is it doesn't need a value function in order to work, and we're going to see exactly what it does instead.
u216 1:08:13 So back to the intuition that we had around having something to compare a reward to, the intuition behind GRPO is given a prompt, you're going to sample multiple responses
u217 1:08:31 in order to get that expected value. So you take some prompt. You take a positive temperature. And when you generate several responses, they hopefully are all different. And you will get some responses that are better than others. And if you do that, and if you compute
u218 1:08:51 the reward for each of these outputs with respect to the prompts, then if you compare that to the mean of reward and you standardize them, you can get a sense of which response is better to other one with respect to other ones.
u219 1:09:09 And you don't need a value function for that. And of course, you also want to not deviate too much from what you had before. So the same tricks apply. So you have the clipping mechanism which is making sure that you don't have nonzero gradients if you're
u220 1:09:29 too far from the old policy. And you also have that KL divergence term, which allows you to penalize any deviations from the reference model.
u221 1:09:42 Comparison of GRPO and PPO
u222 1:09:42 So it's a lot of math, a lot of formulas. So let's relax for a second and just walk through that illustration together. So how would we go about updating the weights of our LLM if we were to follow GRPO?
1:09:42 · Comparison between GRPO and PPO (opening slide) · A Stanford ICME lecture slide titled "Comparison between GRPO and PPO". It is mostly empty so far: only the row label "GRPO" and a single box containing "q" (the query) appear, which looks like the first step of a diagram being built up. A small inset in the top right shows the lecturer standing in front of a blackboard.
u223 1:10:01 So first, we take a prompt and ask our LLM, which is the policy model, to generate responses. So it would generate one output. It will generate two outputs, actually g outputs. And what you want is to know how good these outputs are.
u224 1:10:24 So each of these outputs with respect to the input would be evaluated with respect to your reward model, which can be a model or something deterministic. And then you would compute advantage values by subtracting the reward with the mean of the rewards
u225 1:10:45 of that group and dividing this with the standard deviation of the rewards. So here you get a relative sense of how each output performs with respect to the group. And what you would do is you would
u226 1:11:03 use these advantages in order to compute your gradient. But you would also make sure to penalize for any deviations with respect to your reference model. So you would also have a KL divergence term along with the clipping.
u227 1:11:23 And you would not have any value function. No value function needed. You only need your policy model. You need the reward model and a reference model. And by the way, the reward model is optional too. Because if you're doing, let's say, reasoning or math or coding, you can leverage verifiable rewards. OK.
u228 1:11:44 So how does that compare with PPO? So PPO, you took some inputs. You put that into your policy model. You generate one response. And what you do is you input that prompt and response into a reward model,
u229 1:12:03 tells you how good how well you're doing. There are many variations of how to incorporate the KL penalty. But one of the variations is to incorporate as part of the rewards. So you would do that.
u230 1:12:24 And in parallel to that, you would use your value model in order to get a token-level prediction of what is the expected final reward if you were to follow that policy. And you would compute an advantage quantity, which
u231 1:12:45 we saw is if we want to be extremely vanilla, it can be the difference between the value functions. But it's actually more complicated than that. And you have a method that does that that is called the Generalized Advantage Estimation, that is called GAE. So you can do that.
u232 1:13:04 And then take your advantage and use that in order to compute the gradient that will update your policy. So I want us to-- so by the way, does that make sense so far? Yeah?
u233 1:13:23 OK. So with that being said, I just want to do a recap of what are the models that are involved and which ones are the ones that we're training versus not. So if you remember, the reference model is fixed. So it's the one that you started from. And you have optionally a reward model,
u234 1:13:43 which you train beforehand and that you're leveraging here. You're not training your reward model. But what you're training is obviously your policy model. And in the case of PPO, you're also training a value model. And you're training it jointly with the policy model.
u235 1:14:05 Because as we saw, the value model tries to estimate the expected rewards if you were to follow the policy. And you are actually training the policy. So you need to make sure that these two are in sync. And that is the core reason why PPO is a hassle, because you need to have all the bandwidth to host the value model
u236 1:14:29 training. And at the end, you obtain these formulas. And I'm glad we didn't start from these formulas, because they can be a little bit intimidating. So something that is similar between the two formulas, the ratios. So here we have the ratio between the probability given
1:14:32 · Comparison between GRPO and PPO objective functions · A Stanford lecture slide places the GRPO objective above the PPO objective. GRPO averages over a group of G sampled outputs per question and adds a KL-divergence penalty to a reference policy. PPO uses a single sampled output and has no KL term in the shown formula. A small inset shows a lecturer standing near a board.
u237 1:14:49 by the policy that we're training over the probability of the policy that is the one that generated the batch, so the old policy. That one is the quantity that you see across both formulas. You also see the clipping.
u238 1:15:07 And here we saw that the clipping was meant for us to make sure that our policy is not going too far from the old policy. And then we also have a KL divergence term. So I mentioned that there are many variations.
u239 1:15:26 In here, GRPO actually expresses the KL divergence in the loss function. But you can very well express it as part of the advantage that you're computing, which is the case of PPO. In terms of the differences, so the previous thing that we saw
u240 1:15:48 was one difference, by the way. So where you put the KL penalty is something that can vary from method to method. The second difference is the way you compute the advantage. So in the GRPO case, what you do is you compute the advantage at the completion level, and you compare
u241 1:16:10 that with the other outputs. And so by the way, if you look at the GRPO formula, you will see that T denotes the token. And you see that the advantage is a function of the output I and the token T. And you may wonder, well why is there
u242 1:16:28 a T if it's at the completion level? You can define the advantage either at the completion level or the token level, which is why there is this more generic formula. But people typically use the completion level 1 just because it's easier, because the token level 1 you need to have some way of distinguishing token level
u243 1:16:49 contributions, which you're typically not doing. So anyway, going back, you see I and T, but it's actually a function of I for GRPO. So you compute that by comparing your reward with respect to the mean of the rewards of your group, and you divide that by the standard deviation.
u244 1:17:09 But for PPO, you're computing the advantage at the token level, and you're using the value function for that. And as a reminder, the value function is also at the token level, and it's trained jointly with the policy. And that's a lot.
u245 1:17:29 So I'll just pause for maybe one minute to see if there is any questions on these two big formulas. Now if you see this slide, does that make sense? Yeah? Yeah?
u246 1:17:46 [INAUDIBLE] So the question is you added in your previous slide the KL divergence for PPO. Yes, I did, but these were more meant to illustrate two types of strategies to penalize. But in the general PPO implementation,
u247 1:18:04 you would actually incorporate that as part of the advantage. And by the way, it's a common thing. Every paper has a different way of implementing this. So don't be surprised. OK.
u248 1:18:22 Cool. Well, everyone is good with these big formulas.
u249 1:18:25 Limitations of GRPO
u250 1:18:25 Great. So let's assume you do that for GRPO. It was actually something that the authors performed on math problems, on reasoning problems. And what they saw was-- so if you remember last week, we said that this was used to incentivize the model to produce chains of reasoning before producing an answer,
1:18:25 · DeepSeek-R1-Zero response length grows during RL training · A Stanford lecture slide titled "Increasing output length" phenomenon, showing a line chart of DeepSeek-R1-Zero's average response length over training steps. The smoothed line rises from roughly 500 tokens at step 0 to about 9,500–10,000 by around step 8,400, with a widening noisy band around it. A small inset shows the lecturer standing beside a blackboard.
u251 1:18:48 and it allowed the performance to increase. And one thing that is also something that the authors saw was that the length of the response tended to increase as you were performing the training. And it even tended to increase even
u252 1:19:08 when the performance of the model was reaching a plateau. So on the left side, the y-axis is the performance of the model for a math problem. And then on the right side, it is the average response length. And so you may wonder, well, why is that happening?
u253 1:19:28 Why would the model incentivize the response from getting longer and longer even when the performance is reaching a plateau? So this has to do with, again, this big formula and how you are performing your normalization.
u254 1:19:48 So here I want you to focus on just one piece of the slide. So do you see the 1 over the response of the outputs? Do you see this term? So when responses are short, this term, is it big or small? [INAUDIBLE]
u255 1:20:12 Big, yes. So when O is small, 1/O is big. How about when O is big, is small. So this coefficient is actually something that is multiplied with the contribution of a token to the objective function.
u256 1:20:32 And here we're saying that if you have a token that happens to be in a short response, then the contribution will be bigger than if it's in a long response. So far, nothing incredible.
u257 1:20:50 But now let's assume you are thinking of it in terms of when the advantage is positive versus the advantage is negative. So if you are in a short output, then if your advantage is positive, it will be a big contribution upwards.
1:20:50 · GRPO objective and the "increasing output length" phenomenon · A Stanford lecture slide shows the GRPO objective function with the per-sequence 1/|o_i| normalization term boxed in red. Below it, a large plus sign under "Short output" and a much smaller plus sign under "Long output" suggest that each token in a short response gets a larger weight than each token in a long one. A small video inset at top right shows a person standing in front of a blackboard.
u258 1:21:13 If you are in a case where it's negative, it will be a big contribution downwards. And if you're in a long output, if it's advantage positive, then it's a slight upward increase. And if you're in a negative advantage output,
u259 1:21:30 it's a slight downward contribution. So what does that mean? So if you look at the cases where you're actually outputting bad answers, relatively speaking, the model will prefer a bad answer
u260 1:21:49 that is longer compared to a bad answer that is shorter. So if we go back here, what that means is when the model comes to a situation where, OK, it cannot do anything about the bad answers, it just generates bad answers,
u261 1:22:08 it will then prefer generating longer, bad answers compared to shorter bad answers. And that is the reason why in this graph, you are seeing a plateau of performance, but the length that is increasing because the model
u262 1:22:26 prefers longer bad responses compared to shorter ones. Does everyone agree with this? And so why is this a problem? Well, it's a problem because you don't want that.
1:22:41 · Mitigating the "increasing length" phenomenon with DAPO normalization · A Stanford lecture slide that contrasts a loss normalization with a per-sequence 1/|o_i| factor (highlighted in red as the problem) against DAPO. DAPO's remedy equalizes token-level contributions by dividing by the total token count across all G sampled outputs. A small video inset shows the lecturer standing in front of a blackboard.
u263 1:22:41 DAPO, Dr. GRPO
u264 1:22:42 And that is one of the motivations where there are some papers that follow GRPO that try to tweak some little parts. This included the normalization term. So here, in the case of DAPO, it wanted to have the same contribution for all tokens. So the normalization is with respect
u265 1:23:00 to the sum of the outputs. And another one GRPO done right, or Dr. GRPO. I don't know how you would pronounce it, but they would just forego of that normalization term. And I want to talk about other problems that can occur. So the one other that I want to mention
u266 1:23:18 is the way the advantages are computed. They are divided by the standard deviation of the rewards. You will have varying standard deviations depending on how complex your input is. And that may introduce a bias. And that's not good.
u267 1:23:36 So you may want to remove the standard deviation. And the second one which I wanted to come back to the point around epsilon is that if you have an action that is of a very small probability, and if you
u268 1:23:56 use the same epsilon for both the lower bound and the upper bound, you're actually not giving a chance for smaller probability outputs to be generated even significantly more. So let's assume your probability is like,
u269 1:24:13 let's say, 0.001 and your epsilon is, I don't know, 0.2, you only are giving a small amount of margin for this to be actually more discovered. And so it is not encouraging diversity. So you may want to choose epsilons that are different for the lower bound and the upper bound.
u270 1:24:35 Cool. OK. I wanted to recap. GRPO, we saw that the gradients are less noisy because they're computing advantages with respect to the group average. And we saw that the magnitude of policy updates are also handled by the regular suspects when
u271 1:24:54 it comes to the methods, so clipping and KL penalty. But we saw that we didn't have explicit token-level credit assignments, and that's not good. We don't have a value function anymore, but we don't have token-level credit assignments.
u272 1:25:14 And we want to solve that. And with that, I'll give it to Shervin.
u273 1:25:17 On-Policy Distillation (OPD)
u274 1:25:20 Thank you, Afshin. So to illustrate this point, we're going to go back to our favorite example that we saw earlier in the lecture regarding a Teddy bear that moves within a sealed box. And we're going to recap what GRPO and verifiable rewards incentivize.
u275 1:25:40 So here, regardless of the reasoning steps, what matters was only the answer. So why is it a problem? As Afshin mentioned, you are going to miss on the intermediary steps. So here I showed what G completions could give
u276 1:25:59 in terms of intermediary steps. So you could have a combination of steps that go both well, steps that go both wrong or just one of the two. And as you can see, every token when you look at the formula is going to be associated to the same end reward.
u277 1:26:20 So you're going to attribute the same advantage as the end, even though the step is wrong, which is one downside. And in the longer run, so you might argue, OK, you might have completions for which the intermediary steps don't make sense, but on average they will.
u278 1:26:40 And on average, the model will learn exactly which ones are good. But the thing is it's highly inefficient because you see, you're not leveraging what you can leverage at each step. So each of your terms of your loss functions might not be correctly rewarding the right things.
u279 1:27:02 So it's noisy locally. And then this phenomenon of learning the correct behavior is one that is based on the long-term. So this is not great. And Afshin just mentioned a few minutes ago that in the GRPO formula you had an advantage that was actually
u280 1:27:23 written out as a function of T. So let's just suppose that we're not following the common practice of being based on outcome supervision. Let's say you're on the process supervision methodology, and you do follow that path. Then as we mentioned, you need dedicated labeling for these token-level advantage to make sense.
u281 1:27:47 And if you look at the paper, they're slicing it at the step level, which is here, what truly makes sense. So you need to label each of them separately. And you might argue if you go back to each step, that you could similarly have an ambiguity of exactly what goes
1:28:03 · What we're missing out on: reasoning paths to answers · A Stanford lecture slide poses the prompt "Teddy is in a sealed box. The box moves to the kitchen. Where is Teddy?" and tabulates four sampled outputs, each with two reasoning steps. Steps are shaded green (correct) or red (flawed). The final answers show that flawed reasoning can still reach the right answer (o2 gives Kitchen), so the answer alone hides reasoning quality. A small inset shows the speaker standing by a blackboard.
u282 1:28:09 wrong in that step. So let's say you talk about Teddy who works in the kitchen. Is the issue that Teddy works or that he walks to the kitchen. So it could be a recursive problem down to the token level. So you would be bound to label token at the ends.
u283 1:28:31 And a reward on each token might be a hard model to train. And as we mentioned here, one of the reasons why we moved away from PPO in the first place is to avoid maintaining the weights of multiple models to train. So this is not what we want to do.
u284 1:28:51 So if I had to recap the benefits that we have in this reinforcement learning stages that we want to keep, so we want on-policy part to stay. So we like to train off of the outputs of the model at stake. Then GRPO had these downside of not
u285 1:29:12 crediting each token individually, but rather the overall outcome and hoping for the best on the things that we learn out of it. And we mentioned just now that we do not want to train any additional model to remedy that. So I wrote down a proposal of what we could do to remedy it. So on-policy training is typically us interested
u286 1:29:38 in the model's own generation. So that is set. One proposal is to train next token prediction regarding the loss. And for the lack of additional training as a constraint, maybe we could look around at any strong models around there
u287 1:29:58 and then see what we could learn from it. So I want to ask you a question. I talked about next token prediction and a model that is stronger than the one at stake. So is there a concept that comes to mind as potentially attractive here?
u288 1:30:21 Yep? [INAUDIBLE] Yeah. So yeah the comment is distillation. Yeah, exactly, because distillation is about next token prediction, transfer of information from one model to the other. Yes, spot on. But not the kind that maybe you know. It's a new one. So before we do that, I'm going to do a quick recap on the last
1:30:37 · KL divergence formula on a Stanford lecture slide · A lecture slide titled "KL divergence" shows the discrete formula KL(P||Q) = sum over i of p_i log(p_i/q_i). The rest of the slide is blank. A small inset in the top right shows a person standing in front of a chalkboard. Stanford University and ICME branding appear on the slide.
u289 1:30:44 that we use to compare distributions of next token predictions, which is the one that we're going to be interested in, so this KL divergence. And I wanted to frame it in very simple terms. So as you know, this loss function is not symmetric. So KL divergence between P and Q is not the same
u290 1:31:06 as the one between Q and P. And if you want a way to know instinctively what it means, so you look at the sum, and you see that you sum based on weights of the probability distribution of the first argument.
u291 1:31:24 So you're supposing here that the world is governed by that probability distribution. And then you have the sum of logs of ratios that will translate in penalization when the probability weights of the two differ. So the way I propose we interpret this, so KL of P and Q
u292 1:31:51 is in a world governed by P. How bad is it to approximate with Q? So typically when you do distillation, you would govern the world with the teacher model. So T would be the first argument, and the student would be the second argument.
u293 1:32:09 But here I said we want to do it on policy with respect to the students. So one natural way to formulate this is not to do KL divergence of T and S, but rather the opposite. And this is called the reverse KL divergence.
u294 1:32:29 And it has some interesting properties attached to it. So when you look at the sum and you look at the distribution of the student and the teacher, you see that when T is not equal to 0 and if S is close to 0,
u295 1:32:52 then the contributions of these terms to the sum are not that big. So we say they are forgiving to these mismatches between these two distributions. But on the other hand, if ti for a given term is small, but SI is nonzero, then you're going
u296 1:33:17 to have this ratio that is going to be very big. And that is associated to a weight that is nonzero. So you're going to get a huge term. And these latter points combined with a typical student model's architecture to not have as many degrees of freedom
u297 1:33:39 as the teacher one to make it be mode seeking, meaning that the peaks in the distribution of the teacher will be those that will be learned by the students. So these did this make sense here, the fact that we use reverse KL divergence?
u298 1:34:01 So I'm asking the question because even though it's what people use today, the original paper did a mix of both. So I think it's not obvious that this is obviously the better one, but-- So now let's go back to our favorite example and apply this way of measuring the loss based
u299 1:34:21 on the distributions of student and teacher. So here what we want to do in on-policy distillation is that we want the student to generate a completion based on a prompt. And then we ask the teacher based on the standard context, how much does the token generated by the students
u300 1:34:42 make sense to you? And we ask this question at each new generated token. So let's say you have this prompt with the problem statement of Teddy in a sealed box. And you ask, where is Teddy? So let's say the sentence starts with Teddy. So Teddy makes sense to the teacher.
u301 1:35:01 I would generate also Teddy. But then the student says, Teddy climbs. And the teacher would say, OK, wait. Climbs is not a high-probability token in my distribution. So this is what the teacher would penalize here and what the reverse KL divergence would penalize and so
u302 1:35:23 on. Do you see anything that is not being penalized here. And that is a surprise to you in this example? So the things that are penalized in this toy example are things in red. Yeah? [INAUDIBLE] So the suggestion is work. So in O2 why is works not penalized either? So do you have an explanation for it?
1:35:23 · On-policy distillation: sampled reasoning outputs for a Teddy puzzle · A Stanford lecture slide on on-policy distillation shows a prompt, "Teddy is in a sealed box. The box moves to the kitchen. Where is Teddy?", and four sampled model outputs (o1–o4) broken into two reasoning steps and a final answer. Outputs o1 and o2 answer "Kitchen", while o3 and o4 answer "Room". The words where reasoning goes wrong (climbs, stays) are highlighted in pink, so the slide shows how errors appear at specific steps. A small video inset shows a presenter standing at the top right.
u303 1:35:45 Or does it make sense that it's not penalized? Yeah? [INAUDIBLE] So yeah, the intuition is great. But the answer was maybe the teacher interprets the Teddy bear as being a human. So as we said, once you generate the next token, all the tokens generated before come into the context.
u304 1:36:07 So let's say you are a teacher model. Now in your context that Teddy can do things like climbing. Then walking might be in the span of possibilities. So it's not something that is super surprising. So that is one property that is interesting. And yeah, that's all I had to say here.
u305 1:36:31 So when we look at on-policy distillation, we see that we have this supervision signal that is dense because it comes at the token level instead of the outcome level. And contrary to SFT, as the name suggests, on-policy, meaning that you're training the model
u306 1:36:54 to be better at with respect to its own completions. And you see in practice great results coming from it. And when you go even further than that, with the same amount of compute, you see that you get more bang out of your buck from this method. And also it is more friendly to parameter constraint situations.
u307 1:37:20 So there is this great blog post by Thinking Machine that did some toy ablation studies. And for example, if you play with the rank of the weights that you want to tune and you do SFT first and on-policy distillation next, you see that the delta between the performance that you
u308 1:37:42 get after SFT is shortened greatly after you apply on-policy distillation, which shows that the method is great at operating under compute constraints. OK, great.
u309 1:38:01 And also, since there is no reward model, there is nothing to hack. And if there was something to hack, it would be more than happy to. Because if on-policy decision wants to bring the loss down to 0, then you would align with the teacher distribution, which is a good edge case. That is great.
u310 1:38:21 But then when I mentioned we need a teacher model, then you need to have that in the first place. Let's say you're training a model, then you need to have that. And you're training a frontier model. Maybe you're the best, so there is no one that could teach you. And then the second thing is that contrary to GRPO
u311 1:38:46 where the signal that you're basing your updates on is true labels, now you're comparing yourself to a teacher models output distribution. And it might not be perfect, nor might it be complete in terms of coverage on the domains
u312 1:39:05 that you're interested in. Which is why it might also come as a limitation. Does that make sense? OK, great. So now I'm going to see each of these limitations more in detail. So do you see a way where we could avoid the need for a teacher?
u313 1:39:27 OK, I know it's not obvious, but once you're going to see it, it sounds magical. So you just need to recall that everything
1:39:32 · Extension: self-distillation with a hinted teacher · A Stanford lecture slide proposing self-distillation: the same student model acts as its own teacher, but is given a hint. An example prompt about Teddy in a sealed box is paired with a hint that gives the reasoning and then says "Now solve it yourself:". A small inset shows the presenter standing in front of a chalkboard.
u314 1:39:32 Self (OPSD) and Multi-Teacher (OPD) variants
u315 1:39:35 that you're inputting for you to obtain the output distributions is done by teacher forcing way. So the figure that Afshin had, I think here between generation and training, you would be on the right side, where once you generate the tokens from your student model,
u316 1:39:56 you put all of them, the prompts, plus the tokens from the student model as input of your teacher. And you look at the output next token prediction of your teacher model. So nothing prevents you from injecting things that could help a potential teacher as part of the prompt,
u317 1:40:17 including the answer and everything that leads up to the answer. So this is what the self-distillation paper does. So you have a prompt. Then you have the complete answer. And you ask the model to come back with an answer once again. And this is just a trick for you to get this nice probability
u318 1:40:39 distributions ready to compare with a student. And the nice thing here is that you can do that with the same model. Does that look magical? Yeah? Yeah, I know, it looks great. OK.
u319 1:40:55 So the second thing that we mentioned is the-- oh, yeah, there was a question? [INAUDIBLE] So the question is, if I had to summarize is, when would you benefit most from on-policy distillation? So yeah, you're right that let's say you're the best model out there, maybe
u320 1:41:13 you will have a lesser benefit from this method. And maybe you would use on-policy decision for other needs. The GLM 5 paper that we saw last week is one example where you would use it to calibrate the fact that you do not lose knowledge that you have learned in the past,
u321 1:41:34 so you would distill yourself on previously gained knowledge. But you're right. You would typically use it on faster models of your series. So let's say you have your frontier lab. You have a pro and a flash model. Maybe on-policy distillation would be what
u322 1:41:52 you use on your flash series. And then your pro series might be more foundational GRPO and then these core techniques that might need more compute but that are based off hardcore labels. Thank you.
u323 1:42:09 Yep. Awesome. So the last thing that I wanted to mention is on the fact that you might be bounded by the teacher's abilities. So one approach is to take a model and then use these verifiable rewards across domains.
1:42:11
u324 1:42:30 And then have experts for each domain. And then when you want to do these on-policy distillation, do it against the expert of that domain. So this is why there is a multi teacher on policy distillation that comes in the picture.
u325 1:42:47 So now is it a surprise to anyone that we need multiple teachers here and not just one? Well, empirically if you do these post-training RL on just a single one, you might have things like forgetting that happens and you might use a given set of parameters
u326 1:43:06 towards multiple objectives. So this is a method that has shown to be more efficient. OK. Awesome. And with that, I hope you have a great weekend. [APPLAUSE]