youtube · talk · 2026-10-15 · Stanford Online

Lecture 3: Training LLMs

The source's own title: Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 3 - LLM training

LLM training progresses from reusable next-token pretraining through targeted midtraining, supervised instruction tuning, preference alignment, and reasoning training. RLHF, DPO, verifiable rewards, and on-policy distillation provide different ways to improve behavior and capabilities, while data quality, computational costs, and capability preservation remain central constraints.

Chosen by: Claude desk (AI agent)

Video1 source

How its claims stand

Borne out by another source: 2
Said here alone: 47

What it says

LLM pretraining learns reusable language patterns through next-token prediction, using teacher forcing, causal masking, and cross-entropy loss. Data quality and mixture selection shape performance. Midtraining adds higher-quality, task-aligned data and extends context, requiring positional adjustments and examples with genuine long-range dependencies. Supervised finetuning teaches instruction following through input/output examples, with loss on response tokens; LoRA reduces adaptation costs through low-rank weight updates. Preference tuning teaches which responses to favor or avoid. RLHF learns a reward model from preferences, then optimizes the LLM against that frozen model while constraining departure from a reference model to limit reward hacking. DPO instead learns directly from fixed preference pairs without a separate reward model. Reasoning training uses verifiable rewards, with GRPO comparing responses to the same prompt. On-policy distillation supplies dense teacher feedback on student-generated sequences. GLM-5 illustrates these stages, including long-context training and separate RL phases. Model-collapse risks underscore careful data curation, while cross-stage distillation helps preserve capabilities that later training may lose.
The lecture reports that the Chinchilla paper recommended roughly 20 training tokens per model parameter and found that many scaled-up architectures were undertrained. u6–7 · 1:33 The ratio is presented as a rough compute-optimal training guideline.
Bits per byte rescales token-averaged cross-entropy by multiplying by the token count divided by the input byte count and by 1/log(2). u47–51 · 14:44 The lecture introduces this normalization to compare losses from models with different tokenizers.
The lecture reports a study in which repeating training data up to four times improved performance in a data-constrained regime. u59–61 · 18:14 The result is presented for a fixed supply of training tokens, not as an unrestricted rule about repetition.
DataComp-LM controls the model-architecture side of training to isolate the effect of dataset quality on performance. u62–64 · 19:17
The lecturer states that the open-weight model GLM-5 was trained on 30 trillion tokens. u68 · 21:31 The figure illustrates the infrastructure demands of large-scale pretraining.
The lecturer gives April 30, 2026 as the training-data cutoff for GPT-6 Astra. u96–98 · 29:39 The example illustrates the lecture's claim that a model does not acquire later events from its original training data.
In the instruction-tuning setup described, the loss penalizes predictions of response tokens, not predictions of the supplied prompt tokens. u113–119 · 34:45 Each response-token prediction is conditioned on the prompt and the response tokens decoded so far.
The lecture suggests hundreds or thousands of SFT examples for a narrow use case and potentially millions for assistant behavior across many tasks. u116–118 · 35:49 These are illustrative dataset scales, and the unit is examples rather than pretraining tokens.
The Bradley-Terry formulation models the probability that response y_i is preferred to response y_j as σ(r_i − r_j), where r_i and r_j are their rewards. u176–178 · 54:44
The reward-model pairwise loss minimizes the negative log sigmoid of the winning response's reward minus the losing response's reward, averaged over the preference data. u187–190 · 58:40
A reward model can use an LLM backbone with a classification head on the last input token's representation, or an encoder-only architecture. u190–194 · 59:52
Reasoning tokens count as model output, and users of the reasoning services discussed are charged for hidden reasoning as well as the visible answer. u246–248 · 1:17:52 Afshine raises this as a budgeting consideration.
Shervine reports that GLM-5 pretraining used a context length of approximately 4,000 tokens, initially emphasizing web data before adding more coding and reasoning data. u290–292 · 1:31:32 The account is presented as a walkthrough of GLM-5's technical report.
The GLM-5 midtraining example uses best-fit packing to keep documents intact, padding unused space rather than freely cutting documents into chunks. The accompanying visualization shows context lengths from 32K to 200K tokens. u307–308 · 1:37:34 The context-length range is shown on the slide; training remains next-token prediction.
Shervine reports that GLM-5 divides RL into reasoning RL for verifiable problems, agentic RL for tasks in an execution harness, and general RL for broader response-quality dimensions. u314–317 · 1:40:02 Examples include math competitions, bug fixes checked by tests, and emotional intelligence.
Train small proxy models on different data mixtures and use their performance to infer a promising mixture for larger-scale training. u64–65 · 20:03
RoPE extension methods such as YaRN adjust frequencies so that long-context position differences fall within ranges encountered during short-context training. u81–85 · 24:43 The adjustment is intended to make learning longer-context relationships easier.
The lecturer recommends applying LoRA to feedforward-network matrices as well as self-attention matrices, citing performance gains from adapting where many model parameters reside. u137–139 · 42:30
For a repeatedly used specialized task, a one-time finetuning update can replace a long prompt that repeatedly instructs the model how to behave. u147–149 · 45:49 The lecturer presents this as a use-case-dependent option for users concerned about costs.
Unexpected LLM behavior should prompt an audit of the SFT data to check whether the training examples and their quality were actually intended. u159–161 · 49:24
When a separate teacher is unavailable, on-policy self-distillation can use the same model as the teacher while supplying that teacher with the correct answer. u274–275 · 1:26:45
Long-context training can address context rot by inserting a fact into the middle of synthetic material and requiring the model to refer back to that fact later. u305–306 · 1:36:50 The proposed exercise targets difficulty remembering earlier facts in long contexts.
Teacher forcing enables parallel next-token training by supplying the true preceding tokens regardless of the model's predictions, while masked self-attention prevents the model from seeing future tokens. u24–32 · 7:12 Training need not generate each preceding token sequentially before predicting the next.
Next-token cross-entropy follows from maximizing the probability of the training sequence: factor the sequence probability into conditional probabilities, take logarithms, negate, and average over tokens. u37–46 · 11:00 Taking logarithms avoids the numerical instability of multiplying many probabilities, and averaging removes sequence-length scaling.
RoPE's lack of learned parameters does not guarantee reliable long-context extrapolation: a model trained only on short sequences has not encountered the rotations associated with longer sequences. u78–80 · 23:44 Long-context training exposes the model to positional relationships absent from short-context training.
A pretrained LLM may autocomplete a question instead of answering it because its training objective has taught text continuation rather than assistant behavior. u99–102 · 30:39 The teddy-bear washing example motivates a separate behavioral tuning stage.
LoRA saves storage across specialized tasks because each task can share the same pretrained weights and store only its own small update matrices. u135–137 · 41:47 Separate fully finetuned models would require storing all model weights for each task.
Pairwise preference judgments are easier for humans to supply than absolute quality scores because comparing two responses is simpler than deciding how good one response is on an absolute scale. u157–158 · 48:43 The ease of comparison motivates collecting preference pairs.
Afshine gives two reasons providers protect reasoning chains: the chains may expose information from users' conversations, and competitors could use high-quality chains to train other models. u241–245 · 1:16:08 Afshine links these concerns to interfaces revealing less reasoning than before.
GRPO generates several responses to a prompt and uses each response's performance relative to the group as its advantage. Training favors higher-advantage responses while penalizing departure from a reference model. u256–260 · 1:20:57 In the verifiable-reward setup described, GRPO does not need a learned reward model.
Shervine explains model-collapse risks through compounded distribution changes: finite samples miss rare human data, models imperfectly approximate the sampled distribution, and preference tuning overweights particular language patterns. u296–301 · 1:33:41 These effects make curation important when training corpora include model-generated text.
GLM-5's final on-policy cross-stage distillation uses teachers from earlier checkpoints to preserve capabilities that later training may have lost. u319–320 · 1:41:46 The teacher checkpoints are taken after SFT and after the first RL stage.
LLM pretraining is a large, reusable stage for learning language patterns, after which the pretrained model can be specialized for particular tasks rather than trained from scratch each time. u14–18 · 3:38
Midtraining adds higher-quality, task-aligned data while retaining next-token prediction; SFT instead uses explicit input/output pairs to steer model behavior. u110–114 · 33:49
LoRA reduces the number of trainable parameters by expressing updates to pretrained weight matrices through low-rank matrices, with roughly similar performance depending on the task and dataset size. u131–134 · 40:18
Preference tuning adds negative signals that pretraining and supervised finetuning do not explicitly provide, allowing an LLM to learn which responses humans prefer and which responses to avoid. u151–156 · 46:40
RLHF first learns a reward model from human preferences, then keeps that reward model frozen while updating the LLM to obtain higher rewards without straying too far from its reference model. u197–204 · 1:01:49
1 source
Also said by
- Video Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs
DPO turns preference learning into a supervised objective without a separately trained reward model. Its loss encourages higher probabilities for preferred answers and lower probabilities for dispreferred answers relative to a reference model. u210–214 · 1:05:38
Reasoning training can use verifiable rewards rather than a learned preference reward model: coding solutions can be tested and math answers compared with ground truth. The lecture's proposed reward also checks that the output contains a reasoning chain. u250–254 · 1:19:12
On-policy distillation combines student-generated responses with token-level teacher feedback. Training penalizes differences between teacher and student predictions along the student's generated sequence. u270–273 · 1:25:12
1 source
Also said by
- Video Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs
Teacher forcing creates a training–inference mismatch: training supplies correct previous tokens, whereas inference uses tokens generated by the model itself. u27–29 · 8:08 The lecturer acknowledges the problem and defers methods for addressing it.
Long-context training requires both substantially more attention computation and examples in which predicting a token genuinely depends on distant information. u91–94 · 27:58 Self-attention has quadratic sequence-length complexity, and suitable long-range training examples are difficult to obtain.
SFT requires exceptionally good behavioral examples and a suitable prompt distribution because the model closely imitates the training data and can become biased toward overrepresented response patterns. u125–126 · 38:49 Generalization and evaluation remain additional challenges.
Maximizing a learned reward can produce reward hacking: an LLM may satisfy an imperfect reward model without producing behavior people actually want. u202–206 · 1:03:33 Penalizing divergence from the reference model limits how far reward optimization can move the LLM.
DPO offers simpler supervised training, but trains on fixed responses that need not come from the current LLM. RLHF is more complex and potentially compute-heavy, but uses the LLM's actual generated responses. u223–226 · 1:10:14 Afshine labels the distinction on-policy versus off-policy training.
Sequence-level correctness rewards provide only coarse feedback on a model's own outputs, whereas conventional token-level supervision is dense but uses sequences the model did not generate. u265–269 · 1:23:32 This trade-off motivates on-policy distillation.
Open-weight model reports can disclose substantial training detail without fully revealing the data recipe. Shervine cautions that alleged distillation from other providers may be part of a data mixture without appearing in the technical report. u321–322 · 1:42:36 Shervine explicitly describes the distillation claims as allegations, not established facts.
Building capable, helpful LLMs requires staged training that combines reusable next-token learning with behavioral supervision, preference or verifiable rewards, and distillation, supported by careful data curation and capability preservation. How do the stages and objectives of LLM training turn a next-token predictor into a helpful model with specialized, long-context, and reasoning capabilities?

How it argues

thesis Staged training builds helpful LLMs while preserving capabilities How do the stages and objectives of LLM training turn a next-token predictor into a helpful model with specialized, long-context, and reasoning capabilities?
1. theme Learning reusable foundations requires curated data Next-token learning creates reusable models, but training efficiency and reliability depend on data quantity, mixture, and provenance.
2. theme Long context must be explicitly learned Reliable long-context behavior requires positional adaptation and training examples that genuinely demand distant information.
3. theme Continuation must become assistant behavior Task-aligned training and carefully designed supervision specialize pretrained models, with low-rank updates reducing adaptation costs.
4. theme Preferences teach what responses to avoid RLHF and DPO add comparative feedback through different optimization procedures, each with limitations.
  1. idea Preference tuning adds explicit negative behavioral signals
5. theme Verifiable rewards train reasoning Automatically checked outcomes enable reasoning reinforcement learning, including group-relative optimization.
  1. idea Verifiable outcomes can replace learned reasoning rewards
    1. mechanism GRPO favors group-relative success while limiting model drift
  2. example GLM-5 separates reasoning, agentic, and general reinforcement learning
6. theme Distillation improves feedback and preserves capabilities On-policy distillation supplies dense feedback on student-generated sequences and can recover capabilities lost during later training.
  1. tension Own-output rewards are coarse; conventional supervision is off-policy
    1. action Apply token-level teacher feedback to student-generated sequences
      1. example Answer-conditioned self-teaching substitutes for a separate teacher
      2. example Earlier-checkpoint teachers preserve capabilities lost during later training

The source, passage by passage

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 3 - LLM training

u1 0:00 Introduction
u2 0:05 Hello, everyone, and welcome to lecture 3 of CME 295. So today is an exciting day because we will learn how to actually train a large language model in a way that aligns with our end goal. But before we do that, as always,
u3 0:25 we will recap what we saw last time. So if you remember, last time, we talked about how we could use the transformer architecture that we had seen in lecture 1 and, in particular, see if there is any component of the transformer that we could leverage for something bigger.
u4 0:46 So we saw that the transformer was composed of an encoder and a decoder. And in particular, if you remove the encoder part, you could have a much simpler architecture that would reframe the problem as a text-to-text problem. And we saw that if we were to scale such an architecture
0:47 · Lecture recap slide on encoder-decoder models at Stanford · A lecturer in a pink t-shirt and glasses stands in front of large chalkboards in a lecture hall, with a projected slide behind him. The slide, titled "Recap of last episode...", shows an encoder block and a decoder block. The encoder is labeled "Compute meaningful embeddings" and the decoder "Generate next token".
u5 1:12 in terms of number of parameters-- so model size, number of tokens that the model is trained on-- we saw a better and better performance. So these were the scaling laws that we saw. But something else that we saw is that, given that we only have a limited amount of compute,
u6 1:33 that in order for us, given some amount of compute, to use it in the best way possible, that we should be careful to not undertrain our model. And in particular, there was a paper, the Chinchilla paper, that saw that actually, roughly speaking,
1:36 · Recap slide: LLM scaling laws from Kaplan and Hoffmann · A Stanford ICME lecture slide titled "Recap of last episode…" with a faded decoder-only diagram on the left (encoder crossed out, "Generate next token") and six scaling-law plots on the right. The top row shows test loss falling as a straight line on log-log axes with compute, dataset size and parameters. The bottom row shows compute-optimal training curves (Hoffmann et al.). A speaker stands in a small inset at top right.
u7 1:54 for every parameter you have to actually train it on 20 tokens. And it found that many of the architectures that were scaled up were actually undertrained. And then, given that having bigger models lead to better performance, we then wondered
u8 2:14 if actually activating all the parameters was actually useful. So we saw a class of models called mixture of experts, which selected specific experts to activate for a given forward pass. And we saw that such an architecture
u9 2:34 was something we could place within the feedforward neural network, which was where most of the compute was happening. And we saw a bunch of other methods that improved the way we use transformer-based architectures since it was first released.
u10 2:52 So in particular, we saw the position embeddings, which was actually something that would be better to happen within the attention layer. So we saw RoPE. We saw some other forms of attention-- so not global, but Sliding Window. We saw a Group Query Attention as a way to save on the keys and values that we need to store in order
u11 3:16 to decode our sequence. And we also saw some normalizing methods. And this is where we left off-- lecture 2. And as I mentioned, this lecture is about learning how we can actually train a large language model.
u12 3:35 So we're going to start with the first step, which
u13 3:36 Pretraining
u14 3:38 is called pretraining. And I want to motivate that step by telling you about how people usually trained models traditionally. So if this class was being held maybe 15 years ago, we would actually try to train models on a per-use-case basis.
u15 3:59 So if, for instance, we wanted to detect spam, we would have one model for spam detection, another model for, let's say, sentiment extraction, another one for, let's say, translation and so on. But the idea here is to actually leverage what we learn in one go and re-use that for other use cases.
u16 4:22 So this is also something that is known as transfer learning. So actually, the idea behind what we want to do is to do a one-time big what we call pretraining to learn about language and then start from there if we want to specialize that for a specific use case.
4:26 · Paradigm shift: from traditional ML to transfer learning · A Stanford ICME lecture slide titled "Paradigm shift". A faded line describes traditional ML as training a model from scratch for each task, while the highlighted line defines transfer learning as reusing a trained model to some capacity. Three stacked boxes (spam detection, sentiment extraction, translation) illustrate models for different tasks, with spam detection highlighted. A small video inset shows the lecturer standing at the top right.
u17 4:43 So this is the idea. We want to have the first step of our training process as being something that learns about language. So it's called pretraining. And we're going to talk more about this. And then the second stage is more about tuning this for a specific use case.
u18 5:02 So in the previous example, let's assume we don't have a general-purpose model but we have one big pretrained model. We would tune that pretrained model for this specific task. And we will actually see that LLMs are general enough to be able to actually tackle a bunch of tasks
u19 5:21 without us having to retrain those, but we will see that more later on. Cool. So as I mentioned, this first step is about learning about language. So what we want to do is to see a bunch of texts, a bunch of codes, a bunch of generated generations,
u20 5:45 all over the internet. So here, typically, we would assemble pieces of text in English, but also in other languages. We would assemble, let's say, codes in Python, in other languages-- math, science, you name it. And we would train our model in a next-token-prediction fashion.
u21 6:09 So I'm going to tell you exactly how that would work. So let's assume we have a piece of text on which we want our model to be pretrained on. So here what we would try to do is to feed some text as input and ask the model to predict the next token,
6:22 · Lecture on LLM pretraining at Stanford ICME · A presenter in a pink t-shirt stands in a lecture hall, gesturing toward a projected slide titled "Pretraining overview" over large blackboards. The slide states the objective function is to predict the next token, illustrated with a sample passage about a teddy bear reading Persian poetry. The goal line above it appears faded, as if mid-animation.
u22 6:32 and repeat that process for all the tokens in the text. So in this example, let's assume we have this piece of text, and we tokenize that with your favorite tokenizer. So here it's an arbitrary way of tokenizing things. And what we would do is we would take our LLM, which, as we saw,
u23 6:53 is typically a decoder-only-based architecture, and we would feed in the first token. And what we would do is try to predict the next token. So here, the very beginning of the prediction task, we only have the beginning-of-sequence token
u24 7:12 that is fed. And here we would try to predict the first token. And then we would input that token to try to predict the next one, and so on and so forth. And actually, you don't need to do that sequentially when you do your pretraining, because it's
u25 7:34 highly parallelizable. So what you would actually do is feed the entire input as inputs and try to predict the next token for all these inputs. So it's highly parallelizable.
u26 7:51 You can think of it as a huge matrix multiplication that is happening under the hood. And it's something that you will also find as being mentioned as being teacher-forcing because here, no matter what the LLM predicts-- so by the way, it's the same thing here.
u27 8:08 No matter what the LLM predicts, we always put as input for the next step which word it should have predicted. So we're forcing the model, given some input, to predict the next token that is in that text. And we saw that this is possible because--
u28 8:29 oh, you have a question. So excellent question. The question is, well, if you teacher-force the model to learn a certain way, it is not actually having things. It is generating itself as inputs. So there is a mismatch between training and inference. So that's an excellent point.
8:29 · Stanford lecture on LLM pretraining with teacher forcing · A presenter in a pink t-shirt stands in a lecture hall gesturing with both hands, in front of a projected slide titled "Pretraining is highly parallelizable via teacher forcing." The slide gives the objective as predicting the next token. It shows a masked self-attention matrix, a triangular yellow and purple grid, and an LLM box fed the input tokens [BOS], A, teddy, bear, is.
u29 8:47 And we will see, actually, how we can solve that later on. So it's a great remark. We will see some methods. Yeah? So the question is, if you give only the beginning of sequence, the LLM has no information, and so do you give a prompt? So the answer is not at this stage.
u30 9:05 But if you don't give anything to your LLM, it will learn to just start the sequence with something that starts the sequence. So it will learn tokens that could be likely to appear at the beginning of the sequence. But to your point, there is no information. So-- yeah.
u31 9:25 Cool. One thing I wanted to add here is that we can do that in a parallel fashion because of a trick that I believe we briefly saw during lecture 1, which is the masked self-attention that allows you to only attend to tokens that
u32 9:46 are before you and yourself. So in other words, if you want to predict, let's say, the next token after "teddy," you would only be able to attend to the token "teddy" and all the tokens before you, thanks to the Masked self-attention.
u33 10:09 Great, and so just formalizing that a little bit more, so what we're doing is we're inputting something into the LLM, which you can think of it as a sequence of tokens, until, let's say, t minus 1. And what you want to do is the LLM to actually produce the probability of the next token
u34 10:31 given your input.
u35 10:33 Cross-entropy loss
u36 10:33 And so in a more formal fashion, you will use the cross-entropy loss in order to do that. So I have a formula on the slide, but I'm going to just derive it from first principles. So as with a usual training setup, what we're trying to do is to learn parameters theta such
10:33 · Pretraining parallelizable via teacher forcing: next-token prediction objective · A Stanford ICME lecture slide titled "Pretraining is highly parallelizable via teacher forcing". It states the goal (learn patterns of language and code, greyed out) and the objective function (predict next token). A diagram shows the input x_<t entering a box labeled LLM, which outputs p_θ(x_t | x_<t). A small video inset shows the lecturer standing at the right.
u37 11:00 that the probability of our model seeing the training data is maximized. So the probability of the sequence-- you can think of it as p of the token-- so x1 until xn. And this probability is something
u38 11:24 that is produced by the model. So it's parameterized by theta. So what you want to do is to find the parameters theta that maximize the probability of a model seeing the training set. So your goal is to maximize this quantity.
u39 11:46 And just going back to the basics of probability, it just turns out that you can have some other way of writing this. So you have p of x1 times p of x2 given x1 times all the way to p of x, let's say, n, given x1 until xn minus 1,
u40 12:17 which you can rewrite with this product. So it's t from 1 to n of p of theta of xt given your input. You want to maximize this quantity.
u41 12:34 The problem with maximizing the products of probabilities, which are each between 0 and 1, is that the product tends to 0, so it can lead to numerical instabilities. So what people do is they typically take the log of this quantity.
u42 12:53 So I'll just do the following. We have-- so let's see-- log of this quantity. So we just take log of this. And this gets transformed to a sum of the log of the probabilities.
u43 13:19 So this is just a property of the logarithm. And then what we do is we want to maximize this quantity. But in machine learning, what we like to do is to minimize things. So we actually minimize the minus of this quantity. And then we just want to not have this quantity be
u44 13:42 a function of the number of tokens in the sequence in terms of scale, which is why we divide by the number of tokens. And this is our loss. And it's a function of theta.
13:45 · Lecture on LLM pretraining and teacher forcing at Stanford · A lecturer in a pink shirt stands in a classroom, gesturing toward a projected slide titled "Pretraining is highly parallelizable via teacher forcing." The slide shows an LLM box that takes the previous tokens x<t as input and outputs p_θ(x_t | x<t), next to the cross-entropy loss formula. Partially erased handwritten derivations of the log-likelihood are on the chalkboard behind him.
u45 14:01 And the theta comes from the p of theta, which is something that our model outputs in the last layer-- so all that to say that you will find the cross-entropy loss in a lot of places. And just instead of memorizing this formula, just think of it from first principles.
u46 14:19 It's just you trying to find parameters theta that maximize the probability of your model seeing the training data. OK, cool. So as I mentioned, p theta is the predicted probability, exceeds your target token x less than t is the past tokens.
14:31 · LLM pretraining via teacher forcing: next-token cross-entropy loss · A Stanford lecture slide explaining that pretraining is highly parallelizable via teacher forcing. The objective is to predict the next token. A box labeled LLM takes the preceding tokens x_<t and outputs p_θ(x_t | x_<t). The cross-entropy loss is the negative average of the log predicted probability over T positions. A small video inset shows a lecturer standing at the right.
u47 14:44 And big T is the number of tokens. So you will sometimes see another loss that is actually the same loss but rescaled called Bits Per Byte, BPB. And the idea is that in the cross-entropy loss that I just
u48 15:05 mentioned, the absolute value is actually a function of your tokenizer. Depending on how you tokenize your data, this value might be different. So in case you want to compare your loss between,
u49 15:25 let's say, a given model that has a certain type of tokenizer and another model that has another kind of tokenizer, things may not be exactly comparable, which is why sometimes you will see this loss expressed in a tokenizer-free manner where you take that loss,
u50 15:47 you actually multiply this by the number of tokens divided by the number of bytes, and you also multiply this by 1 over log of 2, just so that you can interpret this as bits, which is why bits per byte. Cool, OK, so now that we know--
u51 16:08 yeah, you have a question. So the question is, what is B? So B is the number of bytes in your input because each character can be encoded with bytes. So yeah. Yeah?
u52 16:27 So the question is, you use the masked self-attention so that you cannot attend to something in the future. Is it something that is only for decoder or also for encoder? So here in this case, we don't have an encoder, and we only have a decoder. And our goal is to predict the next token. We need to make sure that when we predict the next token,
u53 16:48 we can only rely on the things that we have decoded so far to mimic the inference setup. So long story short is we're in a decoder-only setup. If we were in an encoder-only setup, we could not do this in this fashion. We could not predict tokens one at a time, unless you do something else.
u54 17:07 And we will see that in lecture 8. Yeah. OK, cool.
u55 17:15 Data mixtures
u56 17:15 So now we know how we can train our model, which loss we can use. So here's the cross-entropy loss. The next question is, what kind of data can we actually train our model on? And so here there is a lot of effort that is actually put into the kind of data that you want to train on.
u57 17:34 This data can come from the web. It can come from repositories, from places where there's code. Lately there's an emphasis on being able to solve things with math, with science just in general. The idea is to have a big-sized corpus on which you
u58 17:56 can have your model learn about patterns of language, patterns of codes, and it's typically very big in size. Think of it as trillions of tokens. And I think last time there was a question
18:09 · Pretraining overview: returns on compute when repeating data · A Stanford lecture slide on pretraining, highlighting that repeating data has value. The chart plots final test loss against tokens (epochs): the loss predicted by data-constrained scaling laws (solid) falls more slowly than the loss assuming repeated data equals new data (dotted), and flattens near 2.4 by about 40 epochs. Up to about 4 epochs, repetition is almost as good as new data.
u59 18:14 on whether we would actually repeat the same data if we train it, or if we only were seeing unique tokens. Well, there has been some papers actually looking into this. And there's one paper that is here on the slide, which was looking at the case where you're in a data-constrained regime
u60 18:35 where you only have, let's say, a set amount of tokens. The question is, would you rather stop the training, or is it worth for you actually training multiple times on the same tokens? And the answer in this case is actually yes, up to four times. It's actually something that is allowing
u61 18:55 you to get better performance. So there is this trade-off of whether you would rather see more unique tokens-- is it of high or low quality?-- and whether you want to actually repeat that as well. So that's the question that people ask themselves.
19:14 · Pretraining overview: data mixtures and the DataComp-LM pipeline · A Stanford lecture slide on pretraining, with the "Data mixtures" item highlighted and the goal and objective lines greyed out. A four-step diagram shows the DataComp-LM benchmark workflow: select a scale, build a dataset by filtering or mixing, train a model with a fixed recipe, and evaluate on 53 downstream tasks. A small inset shows the lecturer standing at the right.
u62 19:17 Again, on the topic of figuring out which data to use-- so there's been a lot of work in trying to figure that out. And so I'll just cite this work as well, DataComp-LM, which tries to tease out the impact of having a high-quality data set versus having a better architecture,
u63 19:42 Because two people can try to train their model and wonder if a given data is of high enough quality for their setup. But then the performance could be actually due to their own model architecture. So this is trying to normalize the architecture side so that we
u64 20:03 can only tease out the impact of having a better data versus not. There's a lot of work on that piece as well. And you can also wonder, well, OK, how can I obtain my final data set? Should I put 20% of code, 50% of web, and so on, so on? And so here people have been doing that in multiple ways.
20:10 · Pretraining overview: choosing data mixtures via RegMix regression · A Stanford lecture slide explains the RegMix method for choosing a pretraining data mixture. Small proxy models are trained on different mixes of Hacker News, Github and Philpapers, a regression model is fit to predict target loss from the mix, and new mixes are simulated. The best simulated mixture (22.8% Hacker News, 67.0% Github, 10.2% Philpapers) has a predicted target of 5.34 and is used to train a large-scale model.
u65 20:26 And here one way is to train multiple small models on different mixtures of data and see how each of these small models perform so that we can then infer a potential good mix of data that could work well if we were to scale that up.
u66 20:50 So yeah, I highly recommend also reading that paper in case you're interested. And again, I want to emphasize that the data, preparation data work, is actually an involved one. So I wanted to cite this diagram from Olmol 3, which is an open-weight model.
u67 21:09 And here it just goes through the training pipeline, the data preparation pipeline, which is quite involved. And as I mentioned, the size of the pretraining data set is typically huge, on the order of trillions of tokens, and lately, even tens of trillions of tokens.
u68 21:31 So as an example, GLM-5, which is a relatively recent open-weight model, was trained on 30 trillion tokens, just as a point of reference. So you can imagine that in order to train such a big model, you need to have the right infra in place.
u69 21:53 So I just wanted to say that we will not focus on that in this lecture, but we'll focus on that in lecture 5. So stay tuned if you're interested on that topic. Cool, so this was traditionally the first step that people would do in order to train their LLMs.
u70 22:15 But in recent years, there is a second stage
u71 22:18 Midtraining
u72 22:18 that has emerged which is called midtraining, which aims at injecting knowledge on tasks that are more aligned with your use case. So if, for instance, you care about your model, let's say, being able to reason, what you want to do
22:18 · Pretraining overview lecture slide, Stanford ICME · A Stanford University ICME lecture slide titled "Pretraining overview" with faded body text listing the goal, objective, data mixtures and size of LLM pretraining. A pink banner notes that training systems and infrastructure will be covered in Lecture 5. A small inset at top right shows the lecturer standing in a classroom.
u73 22:37 is to expose your model to more of that code and masked text data. And so here you would use midtraining and rely on higher-quality data, still doing next-token prediction, in order to have your model learn more
u74 22:55 about these use cases. So that's one. And the second thing is training your model to see longer contexts, inputs, because one thing that I did not emphasize here is for pretraining, typically
u75 23:16 for efficiency purposes, we pretrain based on sequence lengths that are on the order of several thousands of tokens. But in reality, you could have tens of thousands of tokens as inputs or even hundreds of thousands of tokens.
u76 23:35 And you need to teach your model to actually be able to handle that. So we're going to spend maybe one or two minutes
u77 23:41 Long context training
u78 23:44 on the long context training. So we saw that this was dependent on our ability to encode positions, which is something that nowadays people do. People use RoPE. We saw that there is no learned parameter within RoPE.
u79 24:04 So we could, in theory, use that out of the box. But the problem is, if you don't do anything, if you just train your model on short sequences and want to actually use that for inference for longer sequences, the reality is that your model actually has never
u80 24:24 seen the rotations of these longer sequences, so it may get confused. So this is the reason why we have this long context training stage where we're trying to expose our model on these longer sequences.
24:32 · Long context training: extending RoPE with YaRN · A Stanford lecture slide on long context training, whose idea is to teach the model to use information over longer distances. It explains that RoPE can be extended, for example with YaRN, and uses a clock analogy (hour slow, minute fast, second very fast) to illustrate rotation at different frequencies. It also shows the block-diagonal rotation matrix R_{θ,m} and diagrams of query and key vectors rotated by angles mθ and nθ. A small inset at top right shows a lecturer at a board.
u81 24:43 And in particular, there are typically some little tweaks that are made to the RoPE frequencies in order to help our model learn better. So in particular, you will see a bunch of extension methods from RoPE.
u82 24:59 So I'll cite one. You may see YaRN. And you may wonder, well, why do you need that? So if you were to simply train your model that you previously trained on shorter contexts, if you were to train that model on long context inputs,
u83 25:20 then you would actually give your model a hard time to learn these longer context position differences, because this would be completely unseen from the model perspective. So what people do is they actually
u84 25:39 interpolate the frequencies in order to have long position differences be within the range of frequencies that were seen during the short context training. So there are some little tweaks on the frequencies
u85 25:59 that are used by RoPE in order to help the model learn better. And in addition to that, there is this concept of curriculum learning, so learning-- yes? So the question is, how do they figure out what is the length that they will encounter?
u86 26:18 So it's preplanned. So they will say, OK-- on the figure, actually, you can see an example from GLM-5. The short context training is done on 4,000 long inputs. And then for the longer context training, it's elongated to 128k or even 200k.
u87 26:38 So they know that number from the start, and they will interpolate accordingly. And then they will use that tweaked frequency and do the training and then let the model learn the weights accordingly. So the question is, how about inference? So they would use the tweaked frequencies
u88 26:57 after the long context training. So they would initially train the model for short context on some frequencies, and then they perform midtraining. They perform long context training in which they tweak a little bit the frequencies. And then they use these frequencies for inference
u89 27:17 and so on. Yeah? So the question is, if you use YaRN, do you completely change your setup? So here you actually rely on the RoPE that you had, but you change the frequencies so that the model is more familiar with the ranges of frequencies encountered
u90 27:37 during long contexts and is more familiar with that, as in, it has seen such frequencies during the short context training. Exactly, yeah. Great.
u91 27:58 So why is this hard? So if you remember, transformer is based on self-attention. Self-attention is everything attends to everything else-- O of n squared complexity. So if you scale up from 4,000 as inputs to hundreds of thousands, well, you have a lot more computation.
u92 28:19 But it's not the only thing. The second thing is it's actually hard to get data on which some parts of the data need some other parts in a long range fashion in order to predict that specific token. So I'll give you an example.
u93 28:37 You need your model to learn to use information far in the past in order to make a prediction far in the future. So you need to have some observation that makes use of something that is far in the past and that is not something that is trivially--
u94 28:57 something that you can come up with. So there is some amount of work that is around coming up with such observations. So there's a number of challenges. But at the end of the day, if you do your pretraining, your midtraining, and your long context training, at the end of that stage,
29:06 · Challenges slide: cost of large-scale model training · A Stanford ICME lecture slide titled "Challenges" with the heading "Cost." and three bullets: the cost is at least millions of dollars, it takes time, and it has environmental/electricity impact. A small inset in the top right shows a person standing near a chalkboard in the lecture room.
u95 29:19 you will, first of all, have spent a bunch of dollars. But then the nice thing is you will have a model that knows about text, that knows about codes, and that has acquired a certain amount of knowledge that was in that data.
u96 29:39 And in particular here, if you look at the model cards of your favorite models on third-party providers, you will see that there is a knowledge cutoff date, which is the date that is used up until which you have your training data.
u97 29:58 So for instance, in the case of GPT-6 Astra, you know that the data that was used to train that model was up till April 30th of 2026. And the problem is that that knowledge is fixed in time. So that means that out of the box,
u98 30:17 if you were to ask your model about something that happened after that cutoff date, your model would have no idea about it because it didn't see it at training. So it doesn't have the knowledge. But we will see techniques to actually circumvent that. So more on that later.
u99 30:39 So I have a question for you. We've done all this work. We've started from our LLM, just did all of that pretraining, midtraining, and long context training. And let's suppose we input the following prompt. "Can I put my teddy bear in the washer?" What is one response that the LLM can give us at that stage?
u100 31:07 Yeah? I think you have the right intuition. So here what it will do is-- well, it will not respond to the question. Typically, it will actually try to autocomplete because the model so far has been trained in a next token prediction fashion. It has been trained to guess the next word.
u101 31:28 So here it will respond something along these lines. So, "Can I put my teddy bear in the washer?" Well, "Teddy bears are often made of materials like polyester and cotton, with plastic eyes and sometimes small accessories"-- typically the kind of thing that you would see in text. Is it useful? No, it's not useful.
u102 31:46 So we're not done.
u103 31:47 Supervised finetuning (SFT)
u104 31:47 This was pretraining. Now let's go into the next stage, which is trying to make our LLM useful. So if we ask it questions, we want answers. So in this stage-- so first, this stage is called SFT, Supervised Fine-Tuning.
u105 32:08 And the goal is to change the model behavior to align with what we wanted to do. So in the previous example, if you want to ask a question, you want an answer, so you want to an assistantlike behavior. So the idea behind this stage is to come up
u106 32:28 with input/output pairs for which you want your model to respond like the output for a given input. So here the objective function will still be next token prediction.
u107 32:45 You want some input, and you want to predict what the next token is. So in particular, when it comes to acquiring a behavior that is like an assistant, you want pairs of instructions and responses.
32:51 · Supervised finetuning (SFT) lecture slide, Stanford ICME · A lecture slide titled "Supervised finetuning" defines SFT as Supervised FineTuning. It gives the idea (change model behavior by tuning weights), the strategy (collect input/output pairs, train with next-word prediction given the input), and a special case, instruction tuning. The "Idea" and "Strategy" text is faded, while the heading and special case are in full contrast. A small video inset at top right shows a person standing near a blackboard.
u108 33:04 So as an example, let's say the instruction can be, "Write a short story about a teddy bear who likes to read poetry." And then you have some poem, let's say. Another one about "explain why a teddy bear is a great friend," and then you have the response. And what you would do here is pass the input, which is the instruction, into your LLM,
u109 33:30 and try to predict the next token based on that input, via the next-token-prediction objective function, which here we saw is, for instance, done via the cross-entropy loss. So to make sure we're all on the same page here, the instruction could be in blue.
33:44 · Instruction tuning overview: next-token objective given the input · A Stanford lecture slide titled "Instruction tuning overview" states the objective function as predicting the next token given the input. A diagram shows an LLM box with input tokens [BOS], Do, X, ., Sure along the bottom and output tokens Do, X, ., Sure, ... along the top. The pink-highlighted tokens "Sure" and "..." mark the response part, which is where the prediction applies. A small inset shows the lecturer standing at the top right.
u110 33:49 And then what we would try to do is to have our model predict the ones in pink, which is the answer. So the setup is slightly different with respect to pretraining. Yes? So the question is, how is SFT different from midtraining?
u111 34:08 So midtraining is about higher-quality data. But you don't have one specific prompt as input that you're trying to just mimic a behavior as output. It is still a corpus of texts for which you're trying to predict the next token. The main difference is in midtraining compared
u112 34:26 to pretraining, you have higher-quality data and data that is more aligned with the task of interest. So if you're interested in, let's say, math problems, you would have more math-related data. But here the idea is specifically to tweak your behavior, the behavior of a model,
u113 34:45 to align with what you want. And in this example, we're passing it an input x. So we're not telling our model to predict the x's as input. And what we want it to do is to predict the next token of the answer. So the slight change with respect to the previous setup
u114 35:06 is the input here is x, the instruction or the prompts, and then whatever answer that has been decoded so far. And what you want to do is to predict the next token, again, with a cross-entropy-like loss.
u115 35:27 So you would have the log of the probability of the target token given the input prompt and the past tokens, typically averaged across tokens. So in terms of the kind of data that you would use, you would use data that is aligned with what you want. So here I mentioned one kind of data, which is assistance-
u116 35:49 or instruction-related data. But you could very well have other kinds of data. So let's assume you want, I don't know, specifically writing about poems in a very specific style. You would have only data about that. And the size of your SFT data, which would be much lower compared to the pretraining one--
u117 36:11 so if, for instance, it is a very specific use case, you would have of hundreds or thousands-- hundreds or thousands. But if you want something bigger, let's say, having an assistant-like behavior on a variety of tasks, you would have maybe millions of examples,
u118 36:30 but much lower magnitude compared to the pretraining stage. So back to our example-- let's assume I am asking my LLM, "Can I put my teddy bear in the washer?" What is the kind of-- yes?
u119 36:49 So the question is, do you penalize predicting the input? The answer is no. You only penalize predicting the output. You don't care about the input. The input is given to you. So from this formula you would have, the sum over t of the p theta of--
u120 37:09 log of p theta of t given x and y. Yeah, x is an input, yes, yeah. So the question is, if you compute the loss of the prompt, would it hurt the performance? So you would not really.
u121 37:26 So it's not your intention. Your intention is to not predict what the prompt is. So it's just not aligned with what you want to do. Yeah. Any other questions? OK, back to the question that I had for you-- so let's assume we have the same inputs as last time,
37:41 · Behavior: teddy bear in washer prompt to LLM · A Stanford ICME lecture slide titled "Behavior" shows a simple flow: the question "Can I put my teddy bear in the washer?" goes into a box labeled "Pretrained + instruction tuned LLM", which outputs a question mark. The slide poses the open question of how such a model will respond. A small inset in the top right shows the lecturer standing in front of a blackboard.
u122 37:48 but this time with an LLM that is not only pretrained, but also instruction-tuned with SFT. What would be the kind of response you would get? No. Yeah, exactly, no, but it will say also some other things. Maybe, "No, it might get damaged,
u123 38:07 try hand-washing instead," which is much better because it answers your question. But maybe you're someone who wants to have an answer that is a bit more friendly, maybe with some specific tone.
u124 38:29 So the next section will be about trying to see how we can improve that answer. But before we do that, I wanted to just go over some challenges of the SFT stage. So the first one is, as I mentioned,
38:41 · Challenges slide from Stanford ICME lecture · A Stanford University slide titled "Challenges" lists five bulleted difficulties, with key phrases in bold. A small inset in the top-right corner shows the lecturer standing beside a chalkboard in the lecture room. The slide appears to concern training or aligning language models, though it does not say so.
u125 38:49 we need very high-quality data because when we train our model on that SFT data, the model will really behave exactly like we told it to. So we really need to make sure that the data is exactly what we want. The other thing is, if we have too much data with respect to one specific piece, the model may
u126 39:09 be biased to answer in a very specific way. So there are some challenges with respect to just making sure we have the right distribution of the prompts that we're tuning it on, along with being able to generalize and actually evaluating it. There's one thing that I want to spend a few minutes on, which is the computational part.
u127 39:32 So as we mentioned, our LLM is typically on the order of billions of parameters or even tens or hundreds of billions of parameters. And your model knows already a lot about text, already knows a lot.
u128 39:51 And what you're doing here is trying to just steer its behavior towards something that you would like the model to behave. So the question is, do you really need to tweak all the parameters, or can you actually get away by just tweaking some of them?
u129 40:14 And this is exactly what people have asked.
u130 40:15 Low-Rank Adaptation (LoRA)
u131 40:18 And this is something that motivates LoRA. So who has heard of LoRA? Yeah? Other people? Great. So LoRA stands for Low-Rank Adaptation. And the idea behind LoRA is you actually probably don't need to update all the weights of your model in order to behave a certain way,
u132 40:36 because you're almost there already. So the idea is you take the learnable parameters that you have, which is typically as a matrix, and you decompose that as your pretrained, already-learned weights, W0, and then some tunable parameters.
u133 41:04 And these tunable parameters, they're actually living in much lower-rank matrices, so here B and A. And it slashes completely the number of parameters that you need to train on. So the good thing is it's much more efficient,
u134 41:25 and it turns out to be actually leading to roughly similar performance, depending on the size of your data and your task, which is great. And also-- let's assume, back to our initial example, you have several use cases, and you, let's say, want to tune your model with LoRA for each specific use case.
u135 41:47 The good thing is that you only need to store the pretrained weights, W0, and then the smaller matrices that are task-specific matrices. So here, for spam, if you train LoRA, you get B and A that are specific to the spam use case.
u136 42:10 For sentiment, you can reuse W0, but then have the B and A tuned specifically for the sentiment extraction task-- same for translation and so on. So in other words, if you didn't use LoRA, you would have to store all the weights for the first one, all the weights for the second one,
u137 42:30 and so on. But here you only need to store the delta. So it's also great for storage purposes. So originally, the authors of LoRA, they mentioned people should do that on the matrices that
u138 42:54 are within the self-attention layer. But since then, people have tried different setups, and they have experimented with different data sets. And the latest recommendation is to actually also leverage the matrices that are within the feedforward neural network,
u139 43:13 because it's actually where all the compute is happening. This is where a lot of the parameters are. So it actually leads to a lot of performance gain. And as a suggested reading, there is a great article, I believe, from Thinking Machine that is worth reading that is at the bottom of the slides.
43:18 · Stanford lecture on Transformers and Large Language Models · A still from a lecture video in a Stanford classroom. A man in a pink t-shirt stands beside a chalkboard, looking at a projected title slide, "Transformers & Large Language Models" (Stanford University, ICME). The slide's agenda is on the right, with "Preference tuning" highlighted. Handwritten probability and loss-function equations are on the chalkboard.
u140 43:31 So I highly recommend taking a look if you're interested. Cool. Yeah? So the question is, let's assume I'm asking a given question. How do I know which model takes the task? Well, the answer here is you would typically not
u141 43:50 have different weights if you want a general-purpose assistant. And we'll see this a little bit later. Typically, you would train one big model and have it have as much knowledge that you want, as much knowledge as you can, and answer these questions. So here it's mostly if you have some very, very specific use
u142 44:11 cases. Let's suppose something extremely custom, and you need, let's say, fine-tuning for that. But for the general case, you typically would not do that. Yeah? So the question is, you said the data that is required for SFT is 10 to the power of 6, which is several order of magnitude
u143 44:33 less than for pretraining. So it's less than pretraining. Why don't you do that? Well, people who want to fine-tune, they're usually people like you and I. They don't have super big hardware. And so we cannot afford having to tune billions and hundreds of billions of parameters.
u144 44:53 So that's one. The second thing is here, your model is already there. It has all the knowledge. It has everything. It just is not as efficient to have to retune everything, considering that you're almost there. So it's also, just when you think about it,
u145 45:12 you don't need all these degrees of freedom to get where you want to be. So the question is, are the models across different companies different after this stage? So the answer is, it depends on the architecture that you chose. It depends on the kind of data mixture
u146 45:29 that they're training this on. And it varies quite a lot from model to model. So yeah, they can be different in some way that is determined by these two factors, typically. So yeah, but the data matters a lot. Cool.
u147 45:49 Now I want to just motivate the need-- oh, yes? So the question is, how often do you update the parameters of your model? Well it depends which use case you have. So I would say if you're in a use case where you want your model to do a very specific task and you care, let's say, about costs,
u148 46:11 you don't want to have a big prompt that always tells your model to behave in a certain way. Maybe you want to tweak the weights of your model once with this fine-tuning process, and then re-use that model many times. That is one example, but it highly depends on your use case,
u149 46:30 I would say. Cool. OK, so far, we've seen how to have
u150 46:37 Preference tuning
u151 46:40 our model learn more about knowledge, about text, about code. We learned about how to steer the behavior of our model towards something that we want-- for instance, answering questions. In this section, we will see how we can actually have our model align with human preference data.
u152 47:02 And in particular, I want to motivate this section with the following example. So let's assume we have our LLM, which we have pretrained and fine-tuned. So it knows how to respond to questions or to instructions. So let's suppose we have "Suggest a new activity I could
u153 47:22 do with my teddy bear." And our LLM says, well, "I'd suggest you actually do not spend time with the teddy bear." Well, it's an answer, but it's maybe not the answer that you wanted to output. What you want is to tell your model that this is not the kind of output that you want.
u154 47:43 So, so far, we've only told our model what to do. We have not told it what to not do, because, so far, we've been only saying, OK, next token prediction. Given this input, I want this. And to your point, you had a great point earlier,
u155 48:00 which is that this data is not something that the model has produced. It is actually modeled. We are forcing that data in. And so this is what we want to resolve here. So in order to resolve that, what we do, the first stage is to collect preference-paired data.
48:13 · Preference tuning: collecting preference pairs to train LLMs · A Stanford ICME lecture slide on preference tuning. The faded top half shows the problem: an SFT model answers a request for a teddy bear activity with a bad reply. The bottom states the idea of collecting preference pairs and training on them, with a good reply marked by a green check and a bad reply marked by a red X. A small video inset shows a presenter standing near a blackboard.
u156 48:21 So here for a given prompts, we would collect the negative piece, which is here, not spending time with your teddy bear, and then a positive example, which is maybe something that you want about them being great companions for sleep and so on. So you would have prompts with positive/negative example.
u157 48:43 And the reason why you would frame that as a pairwise problem is just because it is easier to collect such data. So as a human, it is much simpler for you to figure out that something is better than something else, as opposed to stating that something
u158 49:03 is good in absolute scale. Also, for SFT, I mentioned that the distribution of prompts is typically something that causes the model to be quite sensitive. So here the great thing we will see is that the model is not as sensitive to that aspect.
u159 49:24 And we want to steer the model to not misbehave in some way. So here, for instance, the fact of responding as, I actually do not suggest you do anything, it's maybe something that the model would respond, and you did your SFT correctly.
u160 49:44 But sometimes, the model misbehaving actually tells you something about maybe something that happened in the SFT stage that was not intended. So maybe you fed it data that actually you were not intending to do so. The misbehaving parts is also a stage
u161 50:05 where, looking at your SFT data and just making sure that you have trained your model on data that you intended, with the quality that you intended, it's a good time to do that there. So how do you collect preference data? So you can either have a pointwise scoring system--
u162 50:25 so you have a prompt and then you have several answers. This one is good. This one is not good on an absolute scale. But we said that this is kind of hard to do as humans, to have an absolute judgment. You can have this pairwise preference data, which is exactly what we just mentioned,
u163 50:46 where we have two responses, and then we say which one is better than the other. So this is the pairwise setting. And then the listwise setting is you have a prompt and then you have a list of possible answers. And you just rank those from, let's say, most preferred to least preferred way.
u164 51:06 For simplicity purposes, most of the time, people collect pairwise data when it comes to quantifying what kind of preference you want to inject. So the next question is, how can you know, for data that you don't know outside,
u165 51:27 how can you know that a given response is good or bad? How can you generalize that? And this is what is motivating the first step that we're going to talk about, which is called reward modeling.
u166 51:48 So here what we want to do is to build a model that can tell us whether a response to a given prompt is good or not. If we have such a model, then the idea in the second stage is to align our model to align with good behavior, which
u167 52:15 is quantified by our reward model actually outputting good scores for some answers and avoiding outputs that are associated with lower scores. And this is where RLHF comes into the picture.
u168 52:36 So who has heard of RLHF? Yeah, a lot of you. So RLHF stands for Reinforcement Learning from Human Feedback. And it is a method to align your model with human preferences. So why is it called RLHF?
u169 52:55 Because in the first stage, where you want to learn how to distinguish good from bad, you are training your reward model on human preference data. So this is where the human preference gets modeled. And the second stage is to use the model that
u170 53:16 was trained on human preferences in order to align your model to behave better. So that's the idea. So one thing that I will say is that RLHF, and just, in general, reinforcement learning-based methods,
53:27 · Disclaimer slide: RL math deferred to Lecture 4 · A Stanford ICME lecture slide titled "Disclaimer" warns that reinforcement learning-based methods are technical and mathematically involved. A pink callout says the mathematical derivations for RL-based methods will come in Lecture 4. A small inset in the top right shows a person standing near a blackboard in the lecture room.
u171 53:37 they are quite technical, which is the reason why, today, we'll focus first and foremost on the intuition. And we're keeping all the mathematical part for next lecture. So I hope next lecture you'll come prepared.
u172 53:55 We'll do some math. But this lecture, we will focus more on the intuition. And the reason why we're distinguishing the two is because it's actually quite technical, and there are some details to take care of. With that being said, let's now go through the two steps
u173 54:16 that we mentioned-- so reward modeling and reinforcement learning. So reward modeling, our goal is to learn how to distinguish good answers from bad answers. And here, given a prompt and given some answer,
u174 54:36 we want our model to say, yes, it's a good answer to that prompt, or, no, it's not a good answer. So that is our goal.
u175 54:44 Bradley-Terry formulation
u176 54:44 And in order to do that, people typically use a formulation that is quite specific. It's called the Bradley-Terry formulation. And it is a formulation that models the probability of something being preferred over something
54:44 · Reward modeling with the Bradley-Terry formulation · A Stanford lecture slide titled "Step 1: Reward modeling" defines the Bradley-Terry probability that output y_i is better than y_j. It equals e^{r_i} / (e^{r_i} + e^{r_j}), which is the sigmoid of the reward difference r_i − r_j. A small plot of the sigmoid curve rises from 0 to 1 and crosses 0.5 at x = 0. A small video inset at top right shows a presenter standing in a lecture room.
u177 55:04 else in a very specific manner. So here it's probability of, let's say, yi being preferred over yj, which is equal to the sigmoid of ri minus rj. So we're just saying that yi, there is some reward ri that
u178 55:26 characterizes it, yj there is some reward rj that characterizes it. What we're saying is the probability of yi being preferred over yj is the sigmoid of the difference between the two rewards. That is what we're saying.
u179 55:45 So here, as a reminder-- so I'm not sure if you're all familiar with the sigmoid function. The sigmoid function is equal to 1 over 1 plus exponential of minus x. So it has the shape and the slides. So when it tends to minus infinity, it's 0, and over plus infinity, it is 1.
u180 56:05 And the way we're going to train our reward model is we're going to leverage the pairwise preference data that we have collected. So if you remember, we had mentioned that in order to collect preference data, pairwise was the easiest way to do that.
u181 56:25 So we have for each prompt a winning response and a losing response. So let's assume we have a prompt x. And then we have a response yw, winning response, and a losing response yl.
u182 56:45 And our goal is to train our reward model in order to say, well, the winning is indeed the high reward and the losing is a lower reward. So the way we train our reward model is by using a pairwise loss. So for a given prompt, we would put yw and yl.
u183 57:10 We would compute the reward through our model. And then we would update the weights of the model by using this pairwise loss. So this pairwise loss may be a little bit scary. So I'm going to just motivate that.
u184 57:33 So if you remember for cross-entropy, what we said was, intuitively, you want to learn parameters theta that maximize the probability of your model seeing the training set. And so here the event that you want to model is yw--
u185 57:55 so winning response being preferred over a losing response. So here what you want to maximizes the probability that your model says that, let's say, winning response is preferred over a losing response, roughly
58:04 · Person at blackboard about to write, Stanford lecture · A still from a lecture video shows a person seen from behind, wearing a pink t-shirt, glasses and a wristwatch. They hold chalk up to a large, blank dark-gray blackboard as if about to start writing. A faint "Stanford" watermark is in the lower right.
u186 58:19 speaking. So if you remember, so that piece the Bradley-Terry formulation, what does it tell us? It is equal to sigma of r theta, which is your reward model, of-- so your prompts
u187 58:40 and the winning answer minus the reward model of your prompt and the losing answer. So this is what the Bradley-Terry formulation tells us. So you want to maximize this across all your pairs. But as we said, you like logarithm
u188 59:07 because it is monotonic. And these quantities can be very small and can lead to instability. So you take the log. And actually, in the machine learning, you like to minimize things as opposed to maximizing things. So here you want to maximize this, maximizing the log of this, which is the same as minimizing
u189 59:27 minus the log of this. And if you look at the slide, it's exactly what is written. So you want to minimize minus log of sigma of reward of winning minus reward of losing. That is just a result of the Bradley-Terry formulation.
u190 59:52 So let's assume you do that. So in order to-- yeah? Yeah, the question is, what is e? Yes, it's the expectation over your whole data. So you can think of it as a sum or an average if you want. So the question is, what is the architecture of your reward model? Well, there's been several ways people have been modeling rewards.
u191 1:00:11 One way is to leverage the backbone of your LLM. But your LLM is trained in a next-token-prediction fashion. So what you can do is if you put some inputs, you can take the embedding representation corresponding
u192 1:00:32 to the last token that was input and project that onto some space. You can do that. Another way you can do that is you can use an encoder-only model. So you put everything, self-attention, and just project on the data. So the first method, the good thing is that you can leverage the backbone that you already have. So yeah, you can do either way.
u193 1:00:52 But typically, you can do one of these two. OK, cool. So the label here is human preference because you need that in order to say it's RLHF. And actually, I just realized, yeah,
1:01:00 · Step 1: Reward modeling data for RLHF · A Stanford lecture slide titled "Step 1: Reward modeling" describing the data used: on the order of 10,000 observations, labeled by human ratings. The slide notes that human rating is the source of the "HF" in RLHF. A small inset video in the top right shows a person standing near a blackboard.
u194 1:01:12 this was exactly my next slide. So the kind of model would either be a backbone of LLM with the classification head on the last token that you want to leverage or an encoder-only model. Cool, OK, so that first stage, you
u195 1:01:31 went from just having preference pairs and just wanting to learn good responses from bad responses to actually having a model that tells you, for a given prompt and a given response, whether that is actually a good response or a bad response.
u196 1:01:49 Reinforcement Learning from Human Feedback (RLHF)
u197 1:01:49 So now the second stage, which is the more mathematically involved stage, is the reinforcement learning stage. And here the purpose of that stage is to change the weights of your model so that it aligns with the rewards that are mentioned
u198 1:02:08 by that reward model. So how would you do that? So here, you would have a prompt and ask your model to generate a response. And here, you want to know if this is the kind of behavior you want to promote or demote.
u199 1:02:29 So here what you would do is you would input your input prompts and the response into your reward model-- let's suppose it says, no, it's a bad response-- and use that judgment to update the weights of your LLM. So there's a lot of math involved.
u200 1:02:51 But long story short, the reward model is something that you have trained in stage 1 that is now frozen. The weights that you're changing are the weights of your LLM, which is the LLM that you want to align the behavior for. And the way you do that is by using an objective function that
u201 1:03:13 is typically composed of two parts. The first part is trying to maximize the rewards, but also to not deviate too much from the reference model, because, similar to my earlier comment with LoRA, with respect to the pretrained model,
u202 1:03:33 your model here is already good enough. It knows how to respond to questions. It knows a bunch of things. You don't want to change your weights too much. And the thing is, your reward model is not perfect. Sometimes it will say, oh, it's bad, whereas it's good. So you don't want your model to go too far from your reference
u203 1:03:57 model. So you typically have an additional term that penalizes too big of a difference with respect to the reference. So here, typically, the number of observations is orders of magnitude higher than the reward model. And you would initialize your model at the SFT model
u204 1:04:15 that you obtained previously. And you would have an objective function, which is exactly what I mentioned, a trade-off between two pieces, which are maximizing the rewards, but also not deviating too much from the base model. And there's actually a key term that I want to mention here.
u205 1:04:35 People sometimes say reward hacking. So reward hacking is the fact or the phenomenon of your model trying to please the reward model by going into the direction that the reward model likes without this being necessarily a good thing, if you just
u206 1:04:55 forget about your reward model from your perspective, just because the reward model is not perfect. Yeah? It's a great question. So your question is, how do we update the weights of the model? And we'll need to go into details for that.
u207 1:05:14 So save your question for next week. We'll go into that in detail. But the one thing I want you to remember here is in this stage, the reward model is fixed. We're not updating its weights. We're only updating the weights of the LLM. And we will see next week exactly how we do that.
u208 1:05:37 Cool.
u209 1:05:38 Direct Preference Preference Optimization (DPO)
u210 1:05:38 So we just saw RLHF. You need to train a reward model and then do RL on your LLM in order to update its weights in order to align the behavior with what you want. The problem is, RL is complicated. You need a reward model.
1:05:38 · Direct Preference Optimization (DPO) loss function slide · A Stanford lecture slide titled "Supervised approach with DPO" defines DPO as Direct Preference Optimization and shows its loss function, which rewrites preference learning as a supervised objective. It says DPO needs no separate reward model and works directly on preference data. A small inset in the top right shows a lecturer standing at a board.
u211 1:05:56 A lot of attention is needed to select the hyperparameters and so on, which is why people have tried to formulate this problem in a supervised manner, which is the idea behind DPO, which stands for Direct Preference Optimization.
u212 1:06:15 So we will not go into the math here, but I just want to tell you that the loss function that is derived is something that is actually very similar to the loss function that we obtained for the reward model. If you assume that r is equal to some quantity
u213 1:06:36 times the logarithm of the probability from the model that you're trying to tweak over the reference model, and if you see that, you will also see that this loss function incentivizes your model to increase the probability of generating the winning answer
u214 1:07:01 compared to the reference. And it incentivizes your model to decrease the probability of generating a losing answer compared to the reference. So I'll go through that super quickly. So here what we're saying is something,
u215 1:07:24 let's say, minus expectation of, let's say, log of your sigmoids, and then you have this huge thing, like minus rl-- so your loss is such that you want rw minus rl
1:07:25 · DPO loss formula overlaid on lecture video frame · A lecture still shows a person in a pink t-shirt and glasses, seen from behind, writing on a green-gray chalkboard. The Direct Preference Optimization (DPO) loss is overlaid along the bottom. Two terms inside the sigmoid are boxed in red: the scaled log-ratio of the policy to the reference model for the preferred response y_w, and the same for the dispreferred response y_l.
u216 1:07:51 to be as big as possible. So what does that mean? So if you say r is equal to some quantity times log of probability from the model that you're tweaking to, let's say, generate an answer over probability
u217 1:08:12 of your reference model to generate an answer-- so what this is saying is in order for the difference to be big, which is what your loss is trying to do, it will try to make rw as big as possible. So how can I do that?
u218 1:08:33 It is by having the probability of the model we're trying to tweak be higher than the probability of your reference model for that output given the inputs. So this means that we're trying to increase the probability of the winning answer being generated.
u219 1:08:55 And what you want to do is to decrease rl. How do you decrease rl? You actually decrease the probability of your model generating that losing answer given your input. Yes? So the question is, what is pi?
u220 1:09:13 So here pi is-- we will see next week that this model LLM, we will call it a policy. And when you talk about policy, you typically talk about its predicted probability in terms of pi. So yeah, so this notation is rl-specific. And yeah-- to your point, yes.
u221 1:09:34 OK, great. I just want to say that this formula can be scary. But if you take the Bradley-Terry formulation in mind and interpret the left and right term as a function of the ratio between the model that you're trying to tweak over the reference model,
u222 1:09:54 if you think about that this way, then, hopefully, it makes a bit more sense. Yeah? So yeah, the question is, is the reference model the supervised fine-tuned model? Yes, it is the model that you start from when you perform that training, yeah.
1:10:13 · RLHF versus DPO: ease of implementation and performance · A Stanford lecture slide compares RLHF and DPO for aligning language models. RLHF needs multi-stage training and several extra models, while DPO is supervised learning that needs only the base model. On performance, the slide says there is no consensus: results vary by task and are sensitive to implementation.
u223 1:10:14 OK, cool. Moving on, so you will see and hear RLHF and DPO out there. I just want to tell you a little bit about the differences. So RLHF, as we mentioned, is composed of two stages. The first stage is reward modeling.
u224 1:10:33 The second stage is reinforcement learning. So there is models to train, and it can be compute-heavy, whereas DPO is a supervised learning way to do that. But one thing with DPO is it is not
u225 1:10:55 penalizing the loss based on outputs that the model is producing. It is actually on fixed outputs. So to your point, this problem is still present here. And we will see that in a few slides, that RLHF is an on-policy method, because it is actually leveraging the actual outputs
u226 1:11:17 from the LLM, which contrasts with off-policy methods, which are actually training the model based on fixed outputs that are not necessarily generated by the model. So with that, going back to our example,
u227 1:11:37 now let's assume we have the same inputs. Now it's pretrained. It's instruction fine-tuned. And it's also preference-tuned. Now we will have a response that is maybe more friendly in tone. So here, "Can I put my teddy bear in the washer?" Instead of saying, "No," we will have something that is maybe
u228 1:11:56 more friendly. No, "It's better not to. Your teddy could get hurt! A gentle hand wash is safer." So you can use this method to, let's say, change the tone as an example.
u229 1:12:08 Reasoning
u230 1:12:08 So now I want to talk to you about a topic that has increased in importance in the past two years. And this is the topic of reasoning, which I will define here in a very simple way as the ability to solve a problem.
u231 1:12:27 So just to contrast prompts that are nonreasoning versus reasoning-- so here, if you ask a knowledge-based question, what is the course code for Stanford's Transformers & LLMs class?" Well, it's something that you know or you don't know. It is different from, let's say, a math question.
u232 1:12:47 Let's assume you ask, OK, so your bear was born in 2020. How old is the bear now? You need to think a little bit about what year we're in and what is the difference in year in order to get the answer. So this right example is the kind of prompts that we're trying to improve the model
u233 1:13:08 performance on. And if you remember, at the end of last week's lecture, we saw a prompt technique called Chain of Thought that tried to elicit this behavior from the model by incentivizing it to reveal its reasoning chain
u234 1:13:32 through a few-shot examples. So here we saw that if we told the LLM, well, let's say, on this example, how old is the bear, so in order to get to the answer, I want you to output this chain. Then the LLM can learn to output a reasoning chain before the answer.
u235 1:13:51 The problem with that method is we don't want to always tell the LLM to do that. We want the LLM to do that by itself. And this is exactly what we will see here. So in 2024, in September of 2024, OpenAI, I believe, was the first company to tell about improved performance
u236 1:14:20 on math and coding tasks using this technique. And shortly after that, almost all other frontier labs also had models that used the same strategy. And we will see exactly what that strategy is.
u237 1:14:43 In particular, DeepSeek-R1 was the first open-weight model in January of 2025 to actually go through the how. So that paper is probably the first paper that really explains how reasoning models are trained.
u238 1:15:05 And we will see exactly how we do that here. So if we were in, let's say, 2025, the way you would recognize a reasoning model is because it was written everywhere on the UI. So here, an example of OpenAI-- there was a model called ChatGPT-5 Thinking.
1:15:09 · How to spot a reasoning model in 2025 · A Stanford ICME lecture slide with a screenshot of ChatGPT 5 Thinking answering "My bear was born in 2020. How old is he now?" The model thought for 5 seconds and replied that as of November 3, 2025 the bear is either 5 or 4 years old depending on his birthday. A side "Activity" panel shows its visible reasoning, which marks it as a reasoning model. A small inset shows the lecturer standing at the top right.
u239 1:15:24 So here it's a hint. And then the reasoning chain, because it thought for some seconds, it used to output the actual chain, maybe in a summarized way, and you would have also an option to think more. But fast-forward to today-- these models
u240 1:15:42 are actually everywhere. Almost all models are reasoning models these days. And so the only place where, let's say, if you interact with OpenAI, the ChatGPT and so on, the only place you would see a hint of the model being a reasoning model is the thinking effort. So one thing that I want to say is that the reasoning chain
1:16:02 · Lecture on spotting reasoning models, Stanford ICME · A lecture-hall still shows a man in a pink t-shirt standing beside chalkboards and looking at a projected slide titled "How to spot a reasoning model today". The slide is a ChatGPT screenshot. The user asks "My bear was born in 2020. How old is he now?" and the answer is "5 or 6 years old", depending on whether the birthday has passed in 2026. A "Thinking effort" menu is set to "Extra High". The chalkboards carry handwritten log-probability and preference-optimization equations.
u241 1:16:08 is something that is actually important because it encapsulates the previous input that the user has mentioned to the model. And I'm not sure if you've heard in the news, but a couple of months ago, there
u242 1:16:27 was a paper that showed that, for some third-party providers, you could actually recover the full reasoning chain by playing with the API. And the reason why that paper did a lot of noise
u243 1:16:47 is because these reasoning chains, they can actually contain a lot of information about the user, about the chat, what was mentioned before, which is probably not something that you want to expose, for safety reasons. But also, another reason is to not have the reasoning chain
u244 1:17:07 revealed for other people to train on it, because we will see that having a high-quality reasoning chain can actually improve the performance of your model-- so just to say that in general, the reasoning chain is something that people are trying to protect, to not show,
u245 1:17:29 for these two reasons. And this is the reason why you would see the UI that has changed quite a bit between 2025 5 and now, because right now, actually don't really get the reasoning chain out of the box like you used to. So the idea is to incentivize the model
1:17:47 · Enhancing model reasoning abilities: the standard question-to-answer LLM flow · A Stanford ICME lecture slide titled "Enhancing model reasoning abilities". Under the heading "Until now", it shows a simple pipeline in which a Question goes into an LLM, which produces an Answer. A small video inset in the top right shows a person standing in front of a chalkboard.
u246 1:17:52 to actually output a reasoning chain before getting its answer. So just on the what is what, the reasoning chain is actually considered as a model output. So if you think about output token, you actually need to consider the reasoning as part of it.
u247 1:18:14 So if you were to consider using this reasoning models, and if you have a finite budget, just keep in mind that you're being charged for both what you see, but also, the typically hidden reasoning chain that the model produces. And so here I just highlighted in the slides
u248 1:18:35 the parts in the main providers where this is mentioned. Yeah? Yep. So the question is, can the model decide on what to reason versus not? Yes, typically, it has a way to decide. It's part of the training process. You as a user can also set that, typically by some toggle thinking effort.
u249 1:18:54 But even within that, typically, models are trained to categorize prompts that are maybe worth less effort versus more. So short answer is yes. So moving on, the main question now is how we can train such a model, how
u250 1:19:12 we can incentivize a model to output a reasoning chain before its answer. So the idea is to train our model on rewards, but this time, they're not the rewards that we saw a few slides ago.
u251 1:19:30 They are actually verifiable, because when it comes to math and coding problems, when you get an answer, you can tell if the answer is right or wrong. So I'll give you an example. Let's assume you do a coding task.
1:19:45 · Reasoning-based benchmarks: coding problem and solution · A Stanford ICME lecture slide titled "Reasoning-based benchmarks" describes the coding category as solving a coding problem or fixing a bug. It pairs a plain-language problem about teddy bears with a Python function meant to solve it. A small inset at top right shows the lecturer standing near a blackboard.
u252 1:19:48 You have a solution. How do you know if your solution works? Well, you run through some test cases. Let's assume you want to resolve a math problem. How do you know if it's correct? Well, you just compare your answer with the ground truth? So the nice thing with this setup
u253 1:20:14 is that for every model answer, you can tell if your model had it right or wrong. So you can design a reward that relies on the final answer, and also on the fact that the answer needs
u254 1:20:36 to contain a reasoning chain. And so here you can actually have a reward that is composed of the answer being correct and the answer containing a reasoning chain-- so typically having reasoning tokens.
1:20:56 · GRPO used for reasoning training (Stanford lecture slide) · A Stanford lecture slide with a maroon header reading "GRPO used for reasoning training". The body only defines the acronym GRPO as Group Relative Policy Optimization and is otherwise blank, with a citation to the DeepSeekMath paper at the bottom. A small inset in the top right shows a person standing near a blackboard.
u255 1:20:56 Group Relative Policy Optimization (GRPO)
u256 1:20:57 And the way you would train your model would be using rl algorithms. So we will see next week in-depth one such algorithm that has grown in popularity that is called GRPO. That stands for Group Relative Policy optimization. Don't worry the math.
u257 1:21:15 We'll see it next week. But at a very high level, what we're doing is given a query, given a prompt, we're telling our LLM, which is here, the policy, to output several responses. And we figure out if these responses are correct or not. And what we do is we compute advantages,
u258 1:21:37 which is how much better a given answer is relative to the group. If it's much better, let's have the model do more of that. If not, let's have the model do less of that, and still by penalizing with respect to a reference model.
u259 1:21:58 So again, the objective is maximize the advantages while not deviating too much from the base model. So here it's the same mindset. The main difference here is you're actually relying on verifiable rewards. You don't need a reward model.
u260 1:22:17 So here you're just using the knowledge that you have on the ground truth of a problem and comparing whether the answer that is output by the model is matching the ground truth. Does that make sense?
u261 1:22:38 So again, I want to emphasize next week we'll see the details. If you do that, then you will see that your model actually gets better and better at these reasoning problems. So on the left side, we have the performance of the model as a function of the rl step.
u262 1:22:58 So it's a measure of how far you are in the training process. So here it's getting better and better. And on the right side, you have the empirical length of the reasoning chain, which gets increasingly high. I think I already told you, but lecture 4
u263 1:23:17 is where we'll talk about the details. So here, if we have some reasoning problem, we tell the model, well, now, the model can answer in a way that aligns with what we want.
u264 1:23:31 On-Policy Distillation (OPD)
u265 1:23:32 There is something, before giving it to Shervine, that I want to talk to you about. There is a limitation, core limitation of this approach, which is that you're actually getting a very coarse reward. You're getting a reward for your whole output. It's either the whole output is correct
u266 1:23:53 or the whole output is incorrect. So that is one problem. The second problem is that the methods that we saw that had a token-level signal, they had the limitation of being based on sequences that were actually not
u267 1:24:14 produced by the model. So we call such sequences off-policy. So it's an off-policy method. So it's a dense signal, but it's not produced by the model, which can deviate from actually what the model is generating. So we have plus.
u268 1:24:32 It's a dense signal, but it's not from the model. And on the other side, what we just saw with preference tuning and with how to train a reasoning model is that we're actually leveraging outputs from the model at training time, but we are relying on a sparse signal, which is a reward that
u269 1:24:54 is at the sequence level. We don't have any token-level way of just figuring out what is good, what is not good, at least for GRPO. And so there is a new set of training strategies that are emerging.
1:25:04 · On-policy distillation: teacher per-token probabilities on student trajectory · A Stanford lecture slide (ICME) explains on-policy distillation: fix a student model's behavior using a stronger teacher model's outputs. For the prompt "What is 5 + (2 × 3)?", the student's generated tokens are shaded by the teacher's conditional probability. Low-probability tokens such as "2" (5%) and "is" (15%) are shaded darkest red, marking where the student went wrong, while "7" and "21" are 99%.
u270 1:25:12 And there's one such technique that has been becoming more and more popular, which is called on-policy distillation. So the idea behind this method is to combine the best of both worlds. So given a prompt, your model is generating a response. But what you're doing is instead of relying
u271 1:25:38 on a sparse signal that is across the whole sequence, you're actually asking a teacher model to evaluate every single token that was generated. And you penalize deviations between your model, the student model, and the teacher model.
u272 1:26:00 So at a very high level, here, it's as if you were putting the whole sequence that the student is generating. So think of it as just one at a time, you generate the sequence. And you ask the teacher what its prediction would be. And you compare the prediction of the student and the teacher.
1:26:00 · Combining both worlds with on-policy distillation (lecture slide) · A Stanford ICME lecture slide shows an LLM block taking five input tokens ([BOS], A, teddy, bear, is) with arrows up into it. Arrows then lead from the LLM to five empty light-purple output boxes, one per token position. The outputs are left blank, probably to be filled in during the lecture. A small inset shows the presenter standing near a chalkboard.
u273 1:26:24 And you penalize deviations. So the good thing here is it is on-policy because you're actually using the generation of the student. And it is also dense because you're getting a token-level signal. So you have a few variations of that strategy.
u274 1:26:45 So how do you come up with the teacher? If a teacher is not there, there is a variation called on-policy self-distillation, which is you use the same model for a task, but you tell that teacher model, which is the same model, you tell it the answer.
u275 1:27:03 So you tell it the answer so it knows how to steer its response. And this technique is being incorporated in, I would say, most of the frontier open white models. And as an example here, you will see
u276 1:27:23 that you have setups where you actually have specialized teachers-- for instance, one that is very good at math, one that is very, very good at, let's say, engineering. And then what you do is you perform a dense supervision of the model you're trying to train versus these teacher
u277 1:27:42 models. And with that, I'll give it to Shervine.
u278 1:27:47 End-to-end example
u279 1:27:50 Thank you, Afshine. So in this last part, we're going to focus on a practical use case. We're going to see everything that Afshine mentioned in real life. So there is this website called Artificial Analysis Intelligence Index that I recommend.
u280 1:28:10 In case you don't know about it, it's an aggregator of benchmarks, and it translates into a single number, roughly what counts as intelligence these days. So it's a good way to rank models between each other. And there is this filter called Open Source Models. So usually, these open-source models,
u281 1:28:30 they are slightly behind the true frontier from closed-source models. But the great thing about them is that oftentimes they tell all the secret recipe behind the preparation of these LLMs and their open weights. And so we see this ranking.
u282 1:28:49 We have a few strong models. And there is one in particular that has a great technical report, GLM-- so GLM-5, which came out earlier this year. And this will be one that we will anchor on to see everything that Afshine mentioned into practical terms.
u283 1:29:10 So we're going to go through each stage of training the large language model together, and then we're going to see what they have put into each stage. So the first stage that Afshine mentioned was pretraining, where the main goal is to learn about language. So they put some web data.
u284 1:29:31 So what you use is usually Common Crawl that has about everything on the internet. And a lot of it these days is polluted by AI-generated contents and other things, which is why you have heuristics based on this data that
u285 1:29:51 trims everything down. So you have some papers that go into the reasoning of how you do that. Usually, you have about 1% of your input in the end. So it's a great filter. And then it's a good time to teach the model about main lines of knowledge.
u286 1:30:10 So usually, you have authoritative websites, such as Wikipedia, that give a lot of great things on general knowledge. But you have other facts that might not be on these main websites, where you need more care into curating what goes into your data mix. So Afshine mentioned the reasoning aspect
1:30:27 · Pretraining data mix: coding sources slide · A Stanford lecture slide titled "Pretraining data mix" with the Web section greyed out and the Coding section highlighted. It says coding data comes mainly from GitHub and other platforms, with special care for lower-resource programming languages. A small inset in the top right shows a presenter standing near a blackboard.
u287 1:30:31 as being one that is very important these days. So alongside coding, that is at the foundation of everything that powers today's systems. So when we talk about agents or we talk about them in the second part of the class, these are mostly coding problems that are put at different flavors, so
u288 1:30:53 which is why, as early as pretraining, we want to train next-token prediction on coding data-- so typically GitHub and the likes. And then sometimes you have some languages that you care about that might not be as present in your mix. So in particular, mobile-specific data is an example of one.
u289 1:31:14 So this is where some curation is needed. And then, finally, you use material that has this reasoning-like type of data to augment your training.
u290 1:31:32 So you also have things like papers and math and science-focused. So in terms of numbers, as Afshine mentioned, the pretraining stage usually operates on a low-context-length regime. So GLM-5 was at 4K, about, tokens.
u291 1:31:53 And some orders of magnitude-- so it's divided pretraining into two stages, so first one that is very coarse, where we just want to learn about language structure, where we focus on web data. And then, once this is done, then there is a mix of coding and reasoning
u292 1:32:12 that comes in, alongside still web, but at a lesser extent. And now I'm going to talk about exactly what happens at training time. So there was a question here on what do you actually learn. So here we learn purely next-token prediction. And our goal is not to generate useful sentences.
u293 1:32:34 We just want to learn large concepts about language. So we do not have a nicely-formatted speaking sequence end of sequence batch that we just put it as is. We have a ton of data that we chunk, and then we concatenate documents
u294 1:32:52 that go together at once. So you will see things like documents that are cut and then concatenated. And you don't necessarily start your batch with the beginning-of-sequence token, for example. So you do that in order to fill your context length.
1:33:15 · Pretraining visualization: "Concatenate and chunk" strategy for LLMs · A Stanford lecture slide shows how pretraining data is prepared by concatenating documents and chunking them into fixed-length sequences. A box labeled LLM takes a row of tokens as input and outputs the same tokens shifted by one position, for next-token prediction. The span is marked 4K, and documents are separated by [EOS] and [BOS] tokens. A small video inset shows the lecturer at the top right.
u295 1:33:16 And again, we mentioned a typical order of magnitude is like a few K. And now I want to focus a bit more on what I mentioned on the importance of curating your data. So in the first LLMs that came out in 2022, it was great. The internet was mostly human-generated data.
u296 1:33:41 So you didn't need to worry about the quality of who put data in. And I want to talk about a phenomenon called model collapse. So who has heard about model collapse? Yeah, a few folks. So the thing is, imagine you have a perfect model,
u297 1:34:02 and let's say you have human-generated data out there. Let's say you want to train your model. You will sample a finite number of samples from this larger corpus. But the thing is the distribution that you have out there is very diverse.
u298 1:34:22 So you might have some nonzero probability for things that are super long-tail. And the thing is, since your sampling is finite, you will not capture all these details into training your model, which means that the output of your model is by nature doomed.
u299 1:34:41 So you will not have the same distribution as output. And this is with the assumption that the model is perfect. But on top of that, just imagine the model is an approximator. It will not output the perfect distribution. It is taught to learn. And to add insult to injury-- and you might have seen that
u300 1:35:01 into your own use of LLMs-- you have, at later stages, some preferences that vendors and experts in language insert into the behavior of the LLM. So if I talk to you about things that are load-bearing, byte-identical, that might ring a bell about your experience with LLMs.
u301 1:35:20 These are heavily upweighted in frequency with respect to the true distribution. So this is another problem that adds to this initial observation, which is why all the curation techniques that these papers try to achieve at first become really important.
u302 1:35:41 And then I know there was a question a lecture or two ago about the number of epochs that this kind of data runs on. So usually it's [? Of1. ?] But in this paper, the data that is of higher quality per classifiers might run for a bit more. So the second part, which is midtraining--
u303 1:36:03 so Afshine mentioned it's where you have higher-quality data come into the picture. And in the case of GLM-5, the goal was purely to increase context length, and do it in a way that works, which is why the data mixture is mostly composed longer-form data that
1:36:10 · Lecture hall with "Midtraining" slide at Stanford · A presenter in a pink t-shirt stands in a lecture hall, looking toward a projected slide titled "Midtraining" that states the objective is scaling to longer context. Chalkboards behind hold handwritten math on language-model log-likelihood loss and a preference-optimization objective. The slide footer cites the GLM-5 Team, 2026, and carries Stanford University and ICME marks.
u304 1:36:26 matches the domains that we're interested in. So for example, in the case of code, we are looking at PRs with issues and then code diffs. So these typically occupy much more context length. And then you can afford to put in entire books now and so on. And I had mentioned last time the problem
u305 1:36:50 of context rot, where, at larger context length, the LLM might have trouble to remember a fact that was mentioned earlier on. So this is also a nice place to take that into account, where, in synthetic data, you might want to inject some fact in the middle of it
u306 1:37:12 and then try to refer of it later on. So it's another nice place for doing so. And some orders of magnitude-- so you have a similarly, [? then ?] pretraining several stages-- so a first one that focuses on coding, and then a second one on other longer-form data.
u307 1:37:34 And then the strategy to train these is slightly different. So this paper mentions exactly what's the difference. So here, instead of chunking everything, just putting all to predict the next token-- we are slightly smarter about it. We try to pack documents in a way that they fit and that they're not cut.
u308 1:37:55 So it's called the best-fit packing. And if there is too much left, we may pad the context. And yeah, just like pretraining, it's next token prediction, still. So now we want to make the model a useful assistant.
1:38:09 · Midtraining best-fit packing of sequences for an LLM · A Stanford ICME lecture slide titled "Midtraining visualization" shows the "best-fit packing" strategy. Several documents are packed into one long context window, separated by [BOS] and [EOS] tokens, with [PAD] tokens filling leftover space. The sequence is fed into an LLM box, which predicts the input shifted by one token. The context length is labeled "32K up to 200K".
u309 1:38:17 So this is the SFT stage, where data that is of much higher quality comes in. And then the orders of magnitude of the tokens that go into training just go down much more. So there is an emphasis on general helpfulness. And then, just like Afshine mentioned, today's models, they have seen that focusing on reasoning is high-value.
u310 1:38:38 So there is a lot of reasoning and encoding data injected there. And the paper mentions that there is the strategy of inserting the reasoning that leads to a given tool calls, and that is also great for training the reasoning abilities when the model wants to perform a tool call.
u311 1:39:02 So you will see interleaved reasoning and the presence of think tokens. So I think we'll mention it next time more in detail. So all this reasoning that counts in your [? bill ?] and that you don't see, the reason why you don't see it is that it's between tokens called think tokens.
u312 1:39:20 So it's a way for the model to have a scratchpad of where it needs to go, but it doesn't display it to you. And moving on, this time, we do not do pure next-token prediction. So everything that's about the input-- so let's say you are on a general Q&A use case.
u313 1:39:40 You have an input prompt. You do not try to guess the next token there because it's a user input. What you want to fit a next-token prediction on is the answer that you want your LLM to have, which is why the loss is more localized. And then the last stage, RL-- so these days,
u314 1:40:02 RL is heavily used to develop reasoning capabilities. And the GLM-5 paper divides this into three parts. So the first one is on reasoning RL, it calls. So for example, you have these math competitions and other similar kinds of data where you have a place
u315 1:40:24 to think about what the answer is and an answer that is verifiable. So this is the kind of data that reasoning RL is. Agentic RL-- you put your LLM into a harness, where it can use things like running tests to verify that what it does is correct-- for example,
u316 1:40:45 bug fixes. You might want to fix a failing test to a passing test, and so on. And then general RL is a bit more open-ended. So it comes up with dimensions like foundational correctness, emotional intelligence, and task-specific quality,
1:40:47 · Post-training RL stages: reasoning, agentic, general · A Stanford lecture slide titled "Post-training: RL" lists three reinforcement learning stages used to turn a helpful assistant into one that solves tasks reliably. The stages are Reasoning RL, Agentic RL and General RL. A small inset in the top right shows the lecturer standing in front of a blackboard.
u317 1:41:03 which respectively denote the quality of instruction following. Emotional intelligence is the expert data curator's opinion about what a helpful answer is, and so on, so they can inject that opinion there. And in this time-- so we'll see more about it next time.
u318 1:41:26 So we generate the answer. And then, based on whether the answer corresponds to what we're looking for, we might want to upweight the probability of some tokens happening versus not. And I want to connect to the last topic that Afshine mentioned.
u319 1:41:46 So we have seen many stages that try to do many things. And one thing you might wonder about is, are we sure that everything that we have learned is preserved? And it's a valid concern. It sometimes isn't, which is why, at the last stage of RL,
u320 1:42:09 there is this stage of on-policy cross-stage distillation, where the process that Afshine mentioned is done against teachers that are at previous checkpoints-- so more specifically, the one after SFT, and the one after the first RL stage. So it's to make sure that we haven't lost any capabilities.
1:42:34
u321 1:42:36 And so these open-weight models, they're great. They give technical papers with many details. And some of the secret sauce behind the data isn't always spelled out. And I just want to say that-- keep a critical eye on everything you see,
u322 1:42:54 because you see oftentimes, popping up in the news, some providers that complain about others distilling their data. So this might be also part of the data mixture, but of course not written in the technical report. So there are some allegations of it. And also, I want to mention the case of Pi, the coding agent
u323 1:43:19 that you might know already. So this is a minimal agent that boasts great performance with minimal harness. And the author, who I believe was a precursor into understanding really well how these models can perform great, he understood that all these RL
u324 1:43:41 stages, they focus on developing reasoning capabilities, which enable you to avoid putting in too much context into your harness. So this is one great consequence. And you might see the decrease of the need of harness engineering more and more thanks to that.
u325 1:44:02 And sharing things recently in the news-- so all this reasoning data and training enables advances, such as in the field of math. So very recently, OpenAI released some solutions to open problems, which is great.
u326 1:44:22 And with that, I hope you have a great weekend. [APPLAUSE]

The original

Around it

source

zai-org/GLM-5 · Hugging Face

GLM-5 is a large open model from zai-org, presented as a successor to GLM-4.5 aimed at systems engineering and long-horizon agentic tasks. The Hugging Face page gives usage code for Transformers, vLLM, SGLang and Docker, together with benchmark tables, evaluation settings and local deployment commands. Its message is that GLM-5 is the strongest open-source model on reasoning, coding and agentic tasks and is closing the gap with frontier models.

Article

topic

GLM-5

An open-weight large language model from the GLM family developed by Z.ai (zai-org).

Topic2 sourcesNo primary research yet

This text is written with AI.

Placed in: Now