youtube · talk · 2026-10-09 · Stanford Online

Lecture 2: Large language models

The source's own title: Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

Large language models build on transformers, with decoder-only generation becoming dominant alongside representation-based approaches such as BERT. Their effectiveness depends on balancing model size and training data, using efficient architectures, and choosing suitable decoding methods. Prompting and reasoning techniques can improve responses without changing weights, but longer contexts can impair retrieval and increase costs.

Chosen by: Claude desk (AI agent)

Video

How its claims stand

Said here alone: 59

What it says

Transformer language models support two main approaches: BERT learns bidirectional representations for task-specific fine-tuning, while GPT generates text autoregressively and can address tasks through prompts. Scaling laws connect performance to parameters, training tokens, and compute; compute-optimal training balances model and dataset sizes, while serving costs create additional incentives for smaller models. Sparse mixture of experts expands capacity without activating every parameter, using learned routing and load-balancing losses to prevent expert underuse. Modern architectures incorporate rotary position embeddings, sliding-window attention, grouped-query attention, pre-norm, and RMSNorm to encode position, manage computation and cache memory, and support gradient flow. Generation methods include greedy decoding, beam search, and sampling controlled by candidate selection and temperature. Zero temperature does not guarantee deterministic inference, and syntax-constrained decoding improves structured-output reliability. Context windows include generated tokens, and longer contexts can degrade retrieval. In-context examples and chain-of-thought demonstrations can improve responses without weight updates, but add costs and may encourage overfitting; self-consistency stabilizes answers by voting across multiple reasoning paths.
BERT, introduced in 2018, uses the encoder portion of the transformer and lets all input tokens attend to one another. u17–19 · 4:50
BERT's masked language modeling task corrupts selected input tokens and trains the model to recover the original tokens from context. u26–30 · 8:08 The lecture describes masking as a simplification of the actual corruption procedure.
BERT's next sentence prediction task asks whether two input sentences follow each other in a text, aiming to teach relationships at the document level. u31–34 · 9:50
Encoder-decoder transformers use bidirectional attention over the input, whereas decoder-only transformers restrict each input token to itself and preceding tokens. Both architectures can generate text autoregressively. u51–54 · 16:11
FLOPs measures a total number of floating-point operations, while FLOPS or FLOP/s measures operations per second and thus computational speed. u69–71 · 22:14 The distinction is between an amount of computation and a rate.
The lecture reports an approximate compute-optimal relationship of 20 training tokens per model parameter from the 2022 scaling study. u76–77 · 24:39 The relationship concerns training-compute optimality.
The next-token prediction loss discussed in the scaling-law section is cross-entropy, comparing the predicted distribution with the actual next token. u79–80 · 25:45
A language model assigns probabilities to sequences of tokens. The lecture describes large language models as large in parameter count, training-data volume, and computational requirements. u84–86 · 26:49
A decoder-only transformer removes encoder-decoder cross-attention as well as the encoder. Its remaining backbone consists of causal multi-head attention and a feedforward neural network. u87–89 · 27:49
Dense MoE activates every expert with different weights, while sparse MoE activates only selected experts. Sparse MoE combines the selected experts' outputs using weights supplied by the router. u99–104 · 31:18
In the described MoE auxiliary loss, the fraction of tokens assigned to each expert is treated as constant. Backpropagation updates the router through routing probabilities rather than through the discrete selection itself. u122–123 · 38:21
The lecturer reports that modern MoE designs use tens or hundreds of experts rather than just 8 or 16, allowing finer-grained specialization. The lecture also reports empirical performance gains from finer-grained experts. u130–131 · 41:06
Recent frontier open-source models shown in the lecture have hundreds of experts, with examples of 384 and 256, and typically activate about eight experts per forward pass. Their total sizes range from hundreds of billions of parameters to about a trillion. u131–134 · 41:29 These are examples from a recent screenshot discussed in the lecture, not a claim about every MoE model.
The sinusoidal position encoding shown in the lecture assigns sin(ω_i m) to dimension 2i and cos(ω_i m) to dimension 2i+1, with ω_i = 10000^(−2i/d_model). Lower-index dimension pairs vary more rapidly across positions. u161–166 · 51:16 The displayed formula is attributed to Attention Is All You Need.
T5 position bias adds a learned, per-head bias based on bucketed relative distance to scaled query-key scores before softmax. ALiBi instead makes the attention bias a linear function of position difference. u186–191 · 1:00:01
RoPE rotates queries and keys according to their token positions rather than adding an attention bias. The resulting query-key interaction includes a rotation determined by the relative position n−m. u194–206 · 1:02:21
RoPE extends its two-dimensional rotation construction to higher-dimensional vectors by rotating two-dimensional blocks with index-dependent angles. Afshine Amidi emphasizes that the positional construction does not require learning and can be extended to arbitrary positions. u207–209 · 1:07:11 The claim concerns the positional construction, not a guarantee of model accuracy at arbitrary sequence lengths.
RMSNorm rescales activations without recentering them and uses a learned scaling factor. Afshine Amidi says RMSNorm works just as well as LayerNorm, motivating its use. u247–249 · 1:19:42 The comparison is qualitative; the passage provides no benchmark or measured difference.
Top-k sampling restricts candidate next tokens to the k most probable tokens. Top-p sampling instead selects candidates according to a cumulative-probability threshold. u269–271 · 1:26:51 The transcript's explanation of top-p is brief and verbally incomplete.
Temperature T=1 gives the regular softmax distribution; higher temperature flattens token probabilities, while lower temperature sharpens them. Shervine Amidi describes very high temperature as approaching a uniform distribution and very low temperature as concentrating probability on the highest-scoring token. u275–280 · 1:28:52
The lecture gives roughly one million tokens as the industry-standard context length and roughly 100,000 tokens as a typical output limit. u301–303 · 1:37:21 The lecturer treats these as orders of magnitude and says a newly released, unnamed LLM may signal higher limits.
The lecture's token-count rules of thumb are about three-quarters of a word per token, 500 tokens per page, 100,000 tokens for a book and one million for a large book. u303–304 · 1:38:00 These are approximate mental models, not exact conversions.
According to the lecture, the chain-of-thought paper's authors found that demonstrating reasoning paths in few-shot prompts helped the model learn those paths and improved overall performance. u316–317 · 1:41:42 The passage does not specify the tested models, tasks or size of the improvement.
A decoder-only language model can perform sentiment classification by receiving a sentence together with a question asking whether the sentence is positive or negative. The lecture presents fine-tuning as optional for this approach rather than a required adaptation step. u47–49 · 15:12
A load-balancing auxiliary loss can counter routing collapse by encouraging roughly uniform fractions of tokens to be routed to different experts. u120–121 · 37:37 Routing decisions are made separately for each token, not once for an entire sentence.
For reliably structured output such as JSON, Shervine Amidi recommends guided decoding that restricts available next tokens according to the target syntax rather than relying only on a formatting instruction. u289–292 · 1:33:50 The intended setting is an application where malformed structured output causes downstream parsing failures.
The lecture recommends context compaction and starting fresh chats to limit the risks of accumulating more input context. u308–309 · 1:39:18
BERT's CLS-token embedding can represent an entire input sequence because self-attention lets that token interact with every other input token. The representation can then support tasks such as sentiment classification. u19–22 · 5:39
Larger language models can be more token-efficient: they may reach a given performance level after training on fewer tokens than smaller models. u64–67 · 20:32 Token efficiency does not remove the constraint of a finite compute budget.
The lecture places MoE in transformer feedforward layers because those layers typically contain most of the computation, while attention primarily lets tokens communicate. Each expert is therefore a feedforward neural network. u109–110 · 34:12
Routing collapse is self-reinforcing: experts selected early receive updates and improve, while unselected experts remain unused. Concentrating routing on a small subset of experts reduces the model's effective size. u116–119 · 36:07 This feedback loop motivates an auxiliary loss that encourages broader expert use.
Always-active shared experts can handle computation needed by every token, avoiding duplication of common foundational work across specialized experts. u128–129 · 40:23
The dot product of sinusoidal position embeddings at positions m and n is a sum of terms cos(ω_i(m−n)), so the similarity depends on relative offset rather than the two absolute positions separately. The sum is maximized when m equals n. u167–173 · 53:19 The trigonometric identity explains how absolute-position encodings can express relative-position similarity.
Afshine Amidi argues that adding position embeddings to token embeddings unnecessarily exposes every layer to absolute position. Encoding position directly in attention instead targets the operation that needs to assess relationships between tokens. u183–186 · 58:59 The feedforward network is presented as not needing to know a token's absolute position.
Stacking sliding-window attention blocks lets a token receive information from beyond its immediate attention window through intermediate representations. Afshine Amidi compares the expanding range to receptive fields in convolutional neural networks. u219–224 · 1:10:59 Local attention at one layer does not imply that a token's final representation only contains information from one local window.
Grouped-query attention reduces decoding-cache memory by sharing key and value projections within groups of attention heads. Queries are not the same memory target because each query is used once, whereas earlier keys and values are retained for subsequent decoding steps. u232–239 · 1:15:29 The motivation is the growing storage cost of cached keys and values during autoregressive generation.
Pre-norm applies normalization before a sublayer and then adds the sublayer output to the original input. Afshine Amidi says keeping the residual input outside normalization allows the residual connection to support gradient flow more directly than post-norm. u242–245 · 1:18:13 Post-norm instead normalizes the sum of the input and sublayer output.
In-context learning can improve an LLM's responses without changing its weights: examples of questions and desired answers can teach the response format and potentially the reasoning implicitly. u310–312 · 1:40:00 The lecture presents examples as an alternative to weight tuning.
Self-consistency can stabilize answers, for example in arithmetic, by generating several reasoning paths and selecting an answer through majority voting. u318–319 · 1:42:18 The method aggregates multiple generations instead of relying on a single reasoning path.
The lecture presents two main transformer-based approaches: compute representations and fine-tune them for a task, or frame the task as text-to-text generation. Decoder-only models have become the dominant approach for generating text. u56–59 · 18:01
Language-model scaling laws relate performance improvements to training data, parameter count, and compute. A fixed compute budget requires balancing model size against the number of training tokens rather than simply maximizing model size. u61–74 · 19:23
Sparse mixture-of-experts models allow parameter counts to grow without activating all parameters on every forward pass. u140–142 · 44:31
Afshine Amidi presents RoPE, local attention, grouped-query attention, pre-norm, and RMSNorm as common architectural choices in contemporary LLMs, contrasting them with the original transformer. u249–250 · 1:20:22
Shervine Amidi describes LLM generation as autoregressive: processed tokens provide the context for generating subsequent tokens one at a time. u253–255 · 1:21:11
An LLM's context window includes generated tokens fed back into the input, not just the user's original prompt, so available context is linked to output length. u297–299 · 1:36:01
More input tokens are not always better: context rot makes retrieving a particular piece of information increasingly difficult as an LLM's context grows. u306–309 · 1:38:39
Does training-data quality matter for language-model performance? u80–82 · 26:02 The lecturer acknowledges differing research conclusions, but says quantity matters more for pretraining and quality becomes more important in later training stages.
The 2022 scaling study discussed in the lecture found that many contemporary language models were too large for their training datasets. Under the same compute budget, its smaller model trained on more tokens outperformed its larger model. u74–76 · 23:57 The earlier scaling discussion might suggest that choosing a larger model is always better.
Zero-temperature LLM inference can still be nondeterministic when request batching changes floating-point operation order. Shervine Amidi attributes the variation to failures of batch invariance and the non-associativity of floating-point addition. u281–287 · 1:30:58 Fixed weights, fixed operations, and zero temperature would otherwise suggest identical outputs for identical inputs.
BERT can achieve good downstream performance with relatively little labeled data, but task-specific fine-tuning still requires effort. Text generation is also not obvious from the encoder-only design, although the lecture explicitly says it is possible. u38–41 · 12:19 The lecture postpones explaining encoder-only generation until lecture 8.
A language model optimized for training compute may not be the best choice for production serving. Handling many requests creates an additional incentive to keep the model small. u78–79 · 25:23 Training cost and repeated inference cost favor potentially different design choices.
The scaling-law study discussed earlier used dense language models, so its results do not directly establish the scaling behavior of a different MoE design. The lecturer describes running small, architecture-specific scaling experiments before selecting larger-scale parameters. u134–137 · 42:32 Small-scale experiments provide a scaling relationship for the particular architecture being developed.
Learned absolute position embeddings have no learned entries for positions beyond the training sequence length. Afshine Amidi says deterministic position formulas avoid that limitation and reports that the original transformer authors obtained roughly similar performance with both approaches. u150–153 · 47:23 The motivation is to define position information for longer inference sequences without learning additional position embeddings.
Afshine Amidi describes RoPE query-key dot-product magnitude as tending to decay with increasing positional separation, but explicitly says the decay is not strictly monotonic. u210–211 · 1:08:12 The distance-decay description is a high-level tendency rather than a strict ordering of all token pairs.
Choosing the highest-probability next token does not necessarily maximize the joint probability of the completed sequence. Beam search retains k candidate paths to explore alternatives, but costs more computation and remains deterministic. u257–263 · 1:22:34 Beam search is presented as a limited search rather than exhaustive sequence-probability optimization.
Shervine Amidi argues that varied sampled responses can make chatbots feel less robotic, while deterministic responses are useful when reproducibility matters. He gives repeated summarization and consistent LLM-as-judge ratings as examples of the latter. u264–272 · 1:25:05 The repeated-chatbot-response example is a thought experiment because real request contexts may change.
Few-shot examples can make an LLM overfit to the supplied examples rather than reason more generally, while additional input tokens increase cost and latency. The lecture says practitioners sometimes replace examples with handcrafted instructions. u313–314 · 1:41:02 The lecture contrasts example-specific behavior with the desired emphasis on reasoning.
Effective large language modeling depends on balancing training scale, architectural efficiency, decoding choices, and prompting strategies rather than simply increasing model size or context length. How are transformer-based large language models trained, scaled, architecturally refined, and used to generate effective responses?

How it argues

thesis Effective LLMs require balanced scale, architecture, decoding, and prompting How are transformer-based large language models trained, scaled, architecturally refined, and used to generate effective responses?
1. theme From representations to task generation Transformers support both task-specific representation learning and text generation, with decoder-only models enabling tasks through prompts.
  1. idea Representations and generation offer two task-solving approaches
    1. example BERT uses bidirectional encoder attention
      1. mechanism CLS attention builds a sequence representation for classification
      2. mechanism Masked prediction trains BERT to recover corrupted tokens
      3. mechanism Sentence prediction trains document-level relationships
      4. counterargument BERT still requires adaptation and complicates generation
    2. example Prompted sentiment classification can avoid fine-tuning
    3. mechanism Different attention patterns both support autoregressive generation
      1. mechanism Decoder-only backbones retain causal attention and feedforward layers
  2. idea LLMs model token probabilities at large scale
2. theme Scale requires balancing competing resources Effective scaling balances parameters, training tokens, and compute, while accounting for serving costs and architectural differences.
  1. idea Compute budgets require balancing model size and tokens
    1. argument Larger models can reach performance with fewer tokens
    2. evidence Smaller models trained longer beat undertrained larger models
    3. evidence Study estimates twenty training tokens per parameter
    4. counterargument Serving costs can favor smaller than training-optimal models
    5. counterargument Dense scaling results require architecture-specific checks for MoE
3. theme Sparse capacity needs effective expert routing Sparse MoE increases capacity without activating every parameter, but requires routing and specialization that make that capacity useful.
  1. idea Sparse experts grow capacity without activating every parameter
    1. mechanism Sparse routing selects and weights only some experts
    2. mechanism Experts replace computationally intensive feedforward layers
    3. counterargument Self-reinforcing routing collapse shrinks effective model capacity
      1. action Balance expert utilization with an auxiliary loss
        mechanism Router gradients flow through probabilities, not discrete selections
    4. mechanism Shared experts avoid duplicating common foundational computation
    5. evidence Finer-grained expert specialization yields empirical performance gains
    6. example Frontier MoEs activate about eight among hundreds
4. theme Architecture targets positional and computational efficiency Modern LLMs refine positional encoding, attention, and normalization to improve how transformers represent relationships and use resources.
  1. idea Modern LLMs refine positions, attention, and normalization
    1. argument Position belongs in attention rather than every layer
      1. example T5 and ALiBi bias attention by relative distance
      2. example RoPE rotations make query-key interactions position-relative
        mechanism Blockwise deterministic rotations extend RoPE to arbitrary positions
        counterargument RoPE distance decay is not strictly monotonic
    2. example Stacked local attention expands the effective receptive field
    3. example Grouped queries reduce memory through shared cached keys
    4. example Pre-norm preserves a direct residual gradient path
    5. example RMSNorm rescales without recentering while retaining performance
  2. idea Deterministic positions avoid learned embedding length limits
    1. example Sinusoidal positions use dimension-dependent sine and cosine frequencies
      1. implication Sinusoidal similarity depends on relative positional offset
5. theme Decoding choices determine generation behavior Autoregressive generation requires choices about search, sampling, reproducibility, and syntactic constraints.
6. theme Longer context is not always better Context capacity must accommodate generated tokens, and accumulating more context can undermine retrieval rather than improve responses.
  1. idea Generated output consumes the available context window
    1. example Reported context and output limits differ substantially
  2. idea Context rot makes longer inputs harder to retrieve
    1. action Compact context and start fresh chats
7. theme Prompting teaches without updating weights Examples and reasoning demonstrations can improve responses, but their benefits must be weighed against overfitting, cost, and latency.
  1. idea In-context examples teach responses without changing weights
    1. counterargument Few-shot examples risk overfitting, cost, and latency
    2. evidence Demonstrated reasoning paths improved few-shot performance
  2. idea Majority voting across reasoning paths stabilizes answers

Where the source pulls against itself

Larger language models can be more token-efficient: they may reach a given performance level after training on fewer tokens than smaller models. against The 2022 scaling study discussed in the lecture found that many contemporary language models were too large for their training datasets. Under the same compute budget, its smaller model trained on more tokens outperformed its larger model.
The token-efficiency advantage of larger models does not necessarily translate into better performance under a fixed compute budget.

The source, passage by passage

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

u1 0:00 Introduction
u2 0:05 Hello, everyone, and welcome to lecture 2 of CME 295. So today is an exciting day because we'll be talking about large language models, and in particular, how they relate to the transformer architecture that we saw last lecture. But before we start, we will recap what we saw last time.
u3 0:29 And we will do that at every lecture to see how this connect to what we'll see today. So if you remember, last time we were wondering how models could process text. So models, they don't understand text as it is, they understand numbers.
u4 0:47 So if you remember, one thing that we looked at was, first of all, how to divide our text into indivisible units. And this process is called tokenization. And we saw a few kinds of tokenization algorithms. And we said that the most common tokenization algorithm
0:53 · Lecturer in front of "Recap of last episode" slide · A still from a lecture video in a classroom with chalkboards. A man in a dark T-shirt and green pants stands gesturing with both hands while a projected Stanford University / ICME slide behind him shows the sentence "A cute teddy bear is reading." split into tokens. The slide appears to recap tokenization of text, with "teddy bear" kept as one token.
u5 1:06 these days is the subword tokenization because it has a nice trade-off between the vocabulary that it was building and the sequence length that input text would have. But this was not it, because once you
u6 1:23 divide a text into tokens, the next step is to compute embeddings. So if you remember, we saw Word2Vec as a way to compute embeddings, but they were not aware of the context. So then we talked about another class of models that were very popular in the 2010s which is called RNNs,
u7 1:45 Recurrent Neural Networks. And then what we said was, OK, the problem is that you tend to forget things that happened in the past. And so in 2017, there was this landmark paper called "Attention is All You Need" that actually dropped the principle
1:49 · Recap: attention query, keys and values for "teddy bear" · A Stanford ICME lecture slide recapping self-attention on the sentence "A cute teddy bear is reading." The tokens a, cute, teddy bear, is, reading and "." each carry a key (kᵀ) and a value (v). The query for "teddy bear" connects by lines to every token's key. The upper sentence and tokenized row are faded, and a small inset shows the lecturer standing at the right.
u8 2:07 of recurrence and had direct connections in order to compute meaningful representation of tokens. So we spent a little bit of time on some notations. So if you remember, we had query, key, and value, which were useful for us to know OK,
u9 2:28 given a token, what other tokens should that token attend to? And we saw that the architecture that the authors introduced was called the transformer, and that the attention operation that is here is actually something
2:36 · Recap slide: attention and the Transformer architecture · A Stanford ICME lecture slide titled "Recap of last episode..." The right side shows the Transformer encoder-decoder diagram from "Attention Is All You Need" (Vaswani et al., 2017). The left side has a faded illustration of attention on the sentence "A cute teddy bear is reading.", with value, key and query vectors per token and the query for "teddy bear" linked to every key. A small inset in the top right shows a lecturer standing by a blackboard.
u10 2:47 that is happening in the multi-head attention layers, which are present in the encoder and the decoder. And what we said was that this architecture was first developed in the context of machine translation, in that in that context, that model performed really well.
u11 3:11 So a natural question is, well, given that this architecture performs really, really well for machine translation, can we actually propagate the gains to other domains? And so this is what we're going to see in the first part. So we're going to talk about different variants
3:26 · Stanford ICME lecture slide: Transformers & Large Language Models · A title slide from a Stanford University ICME lecture video. The left half is white with the Stanford "S" logo and the title "Transformers & Large Language Models". The right half is crimson with an agenda of five topics, and the first is highlighted. A small inset at top right shows a person standing in front of a blackboard.
u12 3:32 of the original transformer architecture, such that it was possible to apply it to other domains. So if you recall, the transformer is composed of two main parts. The first part is the encoder, which is on the left side of the figure.
u13 3:53 And the goal of the encoder is to compute meaningful embeddings of your input text. So here it was in the context of machine translation. So input text, you can think of it as the source text that you want to translate. And then you had a second part on the right side, which
u14 4:12 is the decoder whose goal is to actually generate the output one token at a time in an autoregressive manner. And the way it does that is by attending to all the tokens that it has predicted so far,
u15 4:33 plus the meaningful embeddings from the input that were computed by the encoder. So that was the original transformer. So a very natural question is, well, how about we try to play with the architecture?
u16 4:47 BERT and encoder-only models
u17 4:50 We try to see, well, what happens if we only take the encoder? Let's suppose we only take the encoder. Can we do something with it? Well, this is something that people have tried shortly after the transformer came out in 2018 with a very important paper called BERT, which stands for Bidirectional Encoder
u18 5:16 Representation from Transformers. So the idea is to take only the encoder parts of the transformer and to leverage the multi-head attention layer which performs the self-attention operation over all of its inputs
u19 5:39 in order to have all the input tokens attend to one another. And so here what they say is, well, if you do that and if you introduce a special token-- and here the special token is on the left side of the sequence called CLS token.
u20 6:00 So it's a classification token. Then what you could do is rely on the meaningful encoded embedding corresponding to the CLS token in order to know what the input sequence is about. Because here the encoder is doing self-attention
u21 6:22 across all the tokens. So the meaningful embedding that is corresponding to the CLS token has interacted with all other tokens. So the idea of BERT is to say if you have some input text, then what I can do is to compute a meaningful representation
u22 6:47 of the input and then use that representation for some downstream task. So at the time we had something like sentiment extraction as a very common task. So is this tweet positive or not? So typically what you could do is take that embedding and then project that onto that class.
u23 7:07 That's one example. But then you could also take the embedding of a given token, and then try to determine if that token is corresponding to let's say, a noun or a verb and so on. So that's the idea.
u24 7:24 Does that make sense? OK, great. So now the next question is, how do you actually train such a model? Well, this is done in two steps. The first step is called pre-training. And the goal is for the model to understand how text is written.
7:27 · BERT and encoder-only models lecture slide · A Stanford ICME lecture slide explains that BERT stands for Bidirectional Encoder Representations from Transformers. Its idea is to use encoded embeddings to make predictions. The diagram shows the tokens "[CLS] A cute teddy bear is reading [SEP]" entering an encoder block, with the [CLS] position projected into a single encoded embedding. A small video inset shows the lecturer standing at the top right.
u25 7:48 And so the way you do that is with some proxy tasks. So if you remember, a proxy task is a task that is not really your end goal, but that is a way for you to achieve something else. And for us, this something else here is the model understanding text.
u26 8:08 So the first proxy task that this model is trained with is called Masked Language Modeling, MLM. And the idea is for the model to understand how tokens are placed.
u27 8:27 And so the way it does that is it takes the input text and then it corrupts it randomly. So the way it does that is it selects some tokens at random and it masks them. So actually, it's a little bit more complicated to not introduce biases.
8:34 · BERT pretraining with Masked Language Model, Stanford lecture slide · A Stanford lecture slide on BERT and encoder-only models, outlining the strategy's first step: pretraining with proxy tasks, specifically the Masked Language Model (MLM). A green ENCODER box sits above a token sequence "[CLS] A cute teddy bear is reading [SEP]", where "cute" and "reading" are shaded grey, marking them as the masked words. A small inset at top right shows the lecturer standing in a room.
u28 8:45 But the idea is to corrupt the input sequence. And the idea is to compute meaningful embeddings corresponding to these corrupted tokens in order to try to predict the actual token-- the original token
u29 9:08 that was before that corrupted token. So here, as an example, if we take a cute Teddy Bear is reading and let's suppose we corrupt cute and we corrupt reading, then here our goal could be to compute a meaningful representation of these corrupted tokens and try
u30 9:28 to predict the original token. And what the model will try to do is to rely on all the attention mechanism that it is performing in order to deduce what the original token was from the context.
u31 9:50 So in other words, the model is trying to understand how language is written and trying to understand, what are the words that are likely to be in places where it is missing? So that's the first proxy task. And then there's a second proxy task that the authors of BERT have also tried, which is called NSP, which stands
10:03 · BERT pretraining with Next Sentence Prediction · A Stanford ICME lecture slide on BERT and encoder-only models. It lists Step 1 of the strategy, pretraining with the proxy task Next Sentence Prediction (NSP). A diagram shows an encoder block fed the tokens "[CLS] Oh wow [SEP] What a cutie [SEP]", with brackets grouping the two sentences. A small video inset shows the lecturer standing at the top right.
u32 10:12 for Next Sentence Prediction. And the idea behind that proxy task is for the model to understand how sentences are sequenced. So here the idea is to have two different sentences as input,
u33 10:33 and to ask the model if these two sentences actually follow each other or not. So in practice, what you would do is you would look at a text and take two sentences that follow each other, put it there, try to predict that it is indeed following. And then two sentences that are not following each other
u34 10:52 and you need to predict that it's not following. And the idea behind that proxy task is for the model to learn text, not at the sentence level, but at the document level. So that's the first step called pre-training.
u35 11:11 And at that stage, you're assuming that your model now understands text much better. And that's why you can go to the second task. So by the way, the first task is typically very expensive, something you do just once, train on a bunch of data. And then the second task is for you
11:23 · Lecture on BERT and encoder-only models at Stanford · A presenter in a dark t-shirt and green trousers stands gesturing in a lecture hall in front of blackboards. A projected slide titled "BERT and encoder-only models" shows the strategy's Step 2, finetuning for a given end task, with an encoder box above the tokens "[CLS] I really love this teddy bear [SEP]". Step 1 (pretraining with proxy tasks) is greyed out.
u36 11:33 to adapt your model to the specific use case that you're interested in. So if we take the example of sentiment extraction, which is trying to predict if a sentence is, let's say, positive or negative, what you would do is have some projection layer that takes in the CLS token that
u37 11:58 is carrying all the information from the sentence and project it onto the class-- the classes of interests which are positive or negative. And for that, you need to have a small set of labels. So you need the observations with actual labels.
u38 12:19 And then what you would do is to tune the weights, so that you can actually project that meaningful representation onto the space of interest. So what are the pros and cons of this approach? So the pro is, well, it led to pretty good performance. The other thing is it is relying on the self-attention mechanism.
u39 12:43 So you have connections from both sides, which is actually the reason why BERT is called that way, because it's Bidirectional Encoder Representation from Transformers. Another pro is, well, you need to do fine tuning, but you don't need that much data typically or of 100.
u40 13:04 But the con is, well, you need a small amount of data to do the fine-tuning, but you need to do the fine-tuning. So in itself, that's an effort. So that's a con. And then the other con is when you look at this model, you may be wondering, well, how can I generate text?
u41 13:26 And it's something that is not obvious. But actually, we will see in lecture 8 that it is actually possible to generate text with an encoder only model. But for now, let's suppose there's no obvious way. But more on that in lecture 8.
u42 13:47 GPT and decoder-only models
u43 13:47 Cool. So that's for encoder only model. Well, going back to the transformer architecture, we tried removing the decoder, only going with the encoder. That's what we obtained. Now how about we remove the encoder and we try with just the decoder?
13:47 · Transformer variant: decoder-only architecture · A lecture slide adapted from the original Transformer diagram, with the encoder block crossed out in red to show that a decoder-only variant drops it. The remaining decoder generates the next token autoregressively. A small inset video in the top right shows a presenter standing in a lecture room.
u44 14:07 Well, let's just try that. Well, actually, people have tried that roughly at the same time as BERT was published. There was another branch of transformer variants which were actually focusing on decoder only transformers. And this is the example of GPT, which
u45 14:27 I'm sure you have heard plenty of times, which stands for Generative Pre-trained Transformer. And the idea is to only keep the decoder parts of the transformer and formulates any task you have as a text-to-text problem. So what does that mean?
u46 14:52 That means you have some text as input and your only job is to predict the next token. So here beginning of sequence A, A, teddy, teddy bear, is, reading, and so on. So by the way, this kind of decoding
14:52 · GPT and decoder-only models lecture slide · A Stanford lecture slide defines GPT as Generative Pre-trained Transformer and states the idea of predicting the next token one at a time. A diagram shows a single DECODER box with a [BOS] token as the input beneath it, the start of an autoregressive generation sequence. A small inset at top right shows the lecturer standing in a classroom.
u47 15:12 is called autoregressive decoding because you predict tokens one at a time. And this is the idea behind GPT. So you don't need to do-- you don't necessarily need to do any fine-tuning because your model is already equipped to be adapted to any use case.
u48 15:33 So I'll give you an example. The sentiment extraction. So let's assume you have a tweet you want to categorize as positive or negative. Well, you would just put into your model, is this sentence positive or negative? And then you put the sentence.
u49 15:51 And then the model would just respond positive or negative. You don't need to do fine-tuning, unless you want to make your model even better. So fine-tuning is not a required step, which is already pretty appealing.
u50 16:10 Encoder-decoder versus decoder-only
u51 16:11 But you may be wondering, well, last time we saw the encoder-decoder architecture, the original transformer, which was also generating text. Well, what is the difference between this new decoder only model with the encoder-decoder model? Well, so both are similar in that they can generate text
u52 16:36 autoregressively. But in the case of the encoder-decoder architecture, the input has a special treatment. It is input into the encoder, and within that encoder, all the tokens are able to interact with one another.
u53 16:57 So it's like pure self-attention. Whereas in the decoder only architecture, what you have as input, the tokens can only interact between themselves and tokens before them. They cannot interact with tokens after them.
u54 17:16 So you have causal attention for decoder only, but you have bidirectional attention for encoder-decoder for the input. That's the first difference. The second difference is that, well, you have an extra encoder and you need to have some special care for the input.
u55 17:40 You're treating it differently, so it may make sense for things like machine translation because, well, you have source texts as inputs and you want to have target text as outputs. But things like text generation, it's not super clear what should be as input.
u56 18:01 So in other words, decoder only is a little bit more general because it treats everything the same. So people have tried both approaches when it came to the text-to-text paradigm, and overwhelmingly they all pretty much
u57 18:21 went with the decoder only. So what I want you to remember is that people have tried all different variants. And in particular when it comes to generating texts, there was both these paradigms. But the decoder only one is the one that actually lived through today.
18:21 · Encoder-decoder versus decoder-only transformer architectures · A Stanford lecture slide contrasts encoder-decoder models with decoder-only models. The encoder-decoder option on the left is greyed out, and the decoder-only option on the right is highlighted. Its diagram shows the encoder crossed out in red, leaving only the decoder, which generates the next token, and the bullets say both input and output use causal attention. A small inset at top right shows a person standing in a lecture room.
u58 18:44 And one of the reasons is, well, it's simpler and there is no special treatment of the inputs. And yeah, you don't need to have an encoder to treat the input separately. So overall, there are two paradigms. Either you compute meaningful embeddings and you try to fine-tuning for your use case,
u59 19:03 or you frame it as a text to text problem. And you can frame all problems as text-to-text problems. Another question is, is it efficient? It's another question. But you can do that. And that's what people have been doing so far. Cool.
u60 19:23 Scaling laws
u61 19:23 Well, as I mentioned, people went with the decoder only architecture, and what they did was just to scale that up, because they have seen that it was performing better and better. And in particular, in the early 2020s, there were a couple of papers that actually quantified how
u62 19:44 much better your model was getting as you were training it on more data, with more parameters, with more compute. And we'll see that now. So this paper from 2020 is actually saying that you can actually have almost like a law that
u63 20:09 tells you how much better your model will get as a function of the number of tokens that you're training on. So it's the middle chart, and as a function of how big your model is, which is the rightmost chart. And so that's the reason why in 2020, 2021, 2022,
u64 20:32 people have been increasing model size and training it on more tokens like increasingly. And not only that, the authors also saw that the bigger your model, the more token efficient they were as well.
u65 20:52 So what does that mean? That means that if you have a bigger model, you need a fewer amount of tokens to train on in order to reach a certain performance. So on the slide, so what are all these colors? So on the y-axis, it's the Tesla's.
u66 21:11 So it's the performance of your model. The lower the better. And then the x-axis is the number of tokens that were processed as part of the training. And so the beautiful colors, they're saying that the brighter you are, the bigger your model is, the more it tends to have a better
u67 21:35 performance earlier. So that's why we're saying that bigger models are also more token efficient. They need fewer training tokens to perform better. But as you can imagine, well, hardware is limited. You don't have infinite compute.
u68 21:55 So very naturally, you're wondering, well, if I have a certain compute budget, how much should I scale my model in size? And how much should I attribute that to the number of tokens I should train it on? So before we talk about this, I want to introduce some notations
u69 22:14 that I think maybe you have seen, but I want to make sure we're all on the same page. So the first one is FLOPs, which stands for Floating Point Operations. And please note the little S, lowercase S. So this is a unit that is meant to measure compute. So the higher the FLOPs, the more compute you actually have.
u70 22:38 So order of magnitude, so training FLOPs for LLMs is like 10 to the power of 25. But you may see another FLOPS out there. So FLOPS with a capital S or even /S. And that one is actually the compute speed. And it's actually a measure of how performant your hardware is.
u71 23:00 It is different compared to the previous FLOPs, but it's easy to get mixed up. So just wanted to call that out. So with that said, back to what we were saying. So we live in a world where we don't have unlimited compute, or at least not yet.
23:00 · Lecturer in front of "Notations" slide on FLOPs · A still from a lecture video shows a man in a dark T-shirt and green trousers standing and gesturing in a classroom with blackboards. A projected Stanford/ICME slide titled "Notations" distinguishes FLOPs (total floating-point operations) from FLOPS or FLOP/s (operations per second). The slide gives rough computational performance for a smartphone, a computer and a supercomputer.
u72 23:18 So given the amount of compute that you have available, a natural question that you have is, how much should I increase my model size? Versus how much should I increase the training size on which I'm training my model?
u73 23:36 And so there is this paper that came out in 2022 which exactly asked this question. And what it found was actually pretty interesting. It found that there was actually a sweet spot between model size and training data size.
u74 23:57 And it also found that most models at the time in 2022 were actually undertrained, meaning that they were too big for the amount of tokens that they were trained on. They were undertrained. And so it actually also showed that by training two models.
u75 24:18 So one, which was, I believe, 70 billion parameters and another one 200 and something billion parameters, so higher than the previous one. And it had the same compute budget for the two. So the first one, given it's smaller in model size, it trained it on more tokens.
u76 24:39 I believe it trained it on $1.4 trillion tokens. And the second one, which was bigger in size, it trained it on fewer tokens because it was the same compute budget. And it showed that the smaller model in terms of size was actually more performant. So they had some relationship between number of parameters
u77 25:01 and number of tokens that you are training it on, which was something like one parameter for 20 tokens. And so this was quite important. A lot of people tried to go back and see OK, is my model compute optimal? So yeah, so this is also pretty important paper.
u78 25:23 Another question you can ask yourself-- I mean, just trying to think beyond this paper is what your use case is. Because here we're talking about training compute optimal. But you also need to serve your model production. So if you serve it with many, many requests,
u79 25:45 you also have an incentive to not have it be too big. But that's another question. But I just want to put it out. Yeah. So the question is, is it the cross-entropy loss? And the answer is yes, because your task is next token prediction.
u80 26:02 And what you're trying to do is have your prediction be as close as the label, which is the next token. And yes, cross-entropy is typically the one that you use and it's the one that is here. Yeah. So the question is, does the quality of the data matter?
u81 26:22 I would say there are many papers that came out from this, many with different conclusions. I would say for pre-training, what matters is the size more than the quality. We will see in lecture 3 that next steps of the training process matters, there's more importance towards quality
u82 26:43 and we will see that. OK, cool.
u83 26:49 Large Language Model definition
u84 26:49 OK, moving on to LLM. So why do we call LLM, LLM? Well LLM stands for Large Language Model. So language model is, because it's a model that associates some probabilities to sequences of tokens.
26:49 · Terminology slide: LLM means Large Language Model · A Stanford lecture slide titled "Terminology" on a mostly blank white background. It gives one definition, LLM = Large Language Model, with the initials L, L and M in bold. A small inset at the top right shows the lecturer standing in front of a board.
u85 27:07 So in this case, it's a model that tries to predict the next token. So we're trying to put some probability towards what the next prediction is. And why is it large? Well, it is large because the model size is large, typically in the billions, even tens of billions or hundreds of billions
u86 27:27 and even trillions. Training data, also hundreds of billions of tokens. At least nowadays, it's also on the tens of trillions of tokens. And also compute. You need a lot of GPUs. And as I mentioned, nowadays people rely on the decoder only transformer,
u87 27:49 which is the original transformer without the encoder. Well, you may ask me, well, if you look into the decoder, you actually have a component, which is the cross-attention that takes in the meaningful embeddings from the encoder with the ones
u88 28:13 coming from the decoder. And you're like, OK, what is happening with that components? Well, you also remove it. So the decoder is not exactly the decoder. You also remove the part that is interacting and you obtain this very simple backbone, which is a causal multi-head attention layer, which is only
u89 28:34 attending to each token and the tokens that came before it and a feed forward neural network, which is where the compute happens. Cool. OK. So what we saw here is the bigger your model size, the better your performance.
u90 28:57 And we saw that there was also a trade-off between the number of tokens you were training it on and the model size that you were having.
u91 29:05 Mixture of experts (MoE)
u92 29:05 A natural question could be, when you have an input x into your model, if your model is huge, then that means that there are a lot of computations happening in that model. A natural question can be do you really
29:05 · Lecture on Transformers and Large Language Models at Stanford · A man in a dark t-shirt and green trousers stands in a classroom, turned toward a projected slide. The slide is the title/outline slide for "Transformers & Large Language Models" from Stanford University and ICME, with the topic "Expert routing" highlighted in bold among six agenda items.
u93 29:24 need all the parameters in your model for that given input x? Do you really need that? And I want to motivate this observation with another observation. So let's assume I'm here in a room full of people.
u94 29:44 And let's say I have a math question. Let's suppose you all are coming from different backgrounds. So let's assume you're a math expert. Let's assume you're a physics expert, biology expert. And I'm coming with my math question. Should I ask everyone?
u95 30:00 Should I only ask you? And this is exactly the kind of question that people have asked themselves. How about we actually only activate a subset of the model for given inputs as opposed to all parameters?
30:07 · Motivation: only a subset of weights is used in the forward pass · A Stanford ICME lecture slide titled "Motivation" states the idea that not all weights are useful in the forward pass. A diagram shows input x flowing into a tall box whose top part, highlighted in blue, is labeled "Subset of the huge model" and whose lower part is faded, then an arrow leading out to the output ŷ. It suggests that only part of a large model needs to be active for a given input. A small video inset of the lecturer stands at top right.
u96 30:22 And so if you were to do that, then you could scale your model in size, but not incur the cost of having to go through all these operations for every forward pass? So how about you did that?
u97 30:39 Well, it turns out there's a type of model out there that was actually predating LLMs, which is called a mixture of experts. So who has heard of mixture of experts? Yes. OK, great. So what is the idea behind mixture of experts? So you have experts.
u98 30:58 So think of it as your math expert, physics experts, et cetera. And then you have a gate, also called a router, that tries to figure out, given your input, which expert should you use and to what extent?
u99 31:18 So for instance, you have some input x and you're passing it through your gate or your router. And the router says OK, expert 2, so this is a math question I should ask you. So there are several kinds of Mixture of Experts or MoEs. One first kind is called a dense MoE.
u100 31:41 A dense MoE, what does it do? It still activates all the experts, but it gives more weight towards experts that are maybe more relevant. So, for instance, the math question could be high weight for the math expert, maybe lower weight for the physics expert, but maybe a very small weight
u101 31:59 for the biology experts. So the bad thing with the dense MoE is you still activate all the parameters. It's not what you want to do. So that's why, in the context of LLMs, the kind that is actually very interesting
u102 32:17 is actually not the dense MoE, it's the sparse MoE, which is the kind of ammo that actually selects experts that should be activated and that ignores all the other experts. This is the sparse MoE.
u103 32:37 So typically you have top k experts that were selected. So k can be equal to 1, can be equal to 2 to some number that you fix. And the output that you get, so y hat is the sum of the expert output weighted by how important they are,
u104 32:58 which is given by the gates. And it's only a sum over some experts that were activated-- so k experts, not all experts.
u105 33:11 MoE-based LLMs
u106 33:11 Cool. Well, OK. All that is great. But now let's go back to reality. Let's go back to our LLM. If you remember, it's a decoder only architecture. So we have this as a backbone. So I have a question for you. Where would you put this mixture of experts scheme in the architecture?
33:11 · MoE in Transformer-based models: decoder block starting point · A Stanford lecture slide titled "MoE in Transformer-based models" shows a single transformer decoder block adapted from "Attention Is All You Need". The block stacks Masked Multi-Head Attention, Add & Norm, Feed Forward, and Add & Norm, with residual connections, repeated N times. The right side is empty, likely where the mixture-of-experts modification will be added. A small inset shows a presenter standing at the top right.
u107 33:33 Which layer would you put it? Does anyone have any suggestions? Yes. Yep. So the answer here is you can take a token and project it in order to select which expert it is.
u108 33:54 Yes. So that is actually the role of the router. So here the thing that I'm really interested in is where would you put the whole mixture of experts in all of this? So I'll help you. Oh, yeah. Sorry. Yeah. It's a great answer. So I'll just exactly mention what you said.
u109 34:12 So the answer would be feedforward neural network, because this is where all the compute typically resides. And yes, I completely agree with you. The multi-head attention layer is actually the place where tokens communicate. It's not where typically you compute meaningful embeddings
u110 34:31 by projecting it. So you would typically go with the feedforward neural network. So a great answer. And so here what you would do is back to your point, have a router where for a given input, decides which expert to activate. And here the expert would be a feedforward neural network.
34:34 · Mixture of Experts in Transformer feed-forward layers · A Stanford lecture slide shows how a Mixture-of-Experts layer replaces the Transformer's feed-forward block. Input x goes to a gating network G, which selects which of the feed-forward experts FFNN1…FFNNn process x. In the diagram, only FFNN2 is selected (green check) and the others are rejected (red X). The outputs combine into ŷ. A faded Transformer block on the left highlights the Feed Forward component being replaced.
u111 34:54 So yeah, great answer. Sorry. Yeah. So the question is, well, or the remark I would say is that in the attention layer, you have different heads. Why wouldn't you treat heads as experts? So it's a great point.
u112 35:14 I think, at the end of the day, you can also try that. But speaking of that, we will actually talk about a small variation that people usually do on multi-head attention because, not all heads are as useful.
u113 35:30 So I'm going to just address your question in about 30 minutes. Yeah, but great point. Yep. So the question is, how does the router know when to activate each expert? So by with the magic of back propagation, magic of learning.
u114 35:50 So you have a training process and then you want to learn how each component should behave. And you just backpropagate and it just learns that way. And it's actually a great point because one observation
u115 36:03 Load balancing loss
u116 36:07 you may tell me is, what would prevent your model from overrelying on a specific set of experts and not explore other experts? And so this is actually a problem when training MoE-based LLMs, which
u117 36:27 is the tendency for your model to rely only on a small set of experts instead of all the experts. And it's called routing collapse. So there are many reasons why it's not good. But one of the reasons is your effective model size is actually smaller than the total number of experts
u118 36:51 and weights that you have. And this typically occurs, because when you train your model, if some experts get chosen in the very beginning, then when it comes to updating the weights, those experts that were activated in the beginning, they
u119 37:13 tend to be updated at first. So they tend to get better and better, whereas all the other experts are still sitting there. So it's the problem of the rich getting richer. And so it's for that reason that people have also come up with a part of the last cult, so it's
37:34 · Routing collapse in mixture-of-experts and auxiliary loss remedy · A Stanford lecture slide on training challenges in mixture-of-experts models. The symptom, routing collapse, is that the same expert is selected most of the time. The remedy is an auxiliary loss, loss = α·N·Σ f_i·P_i, that forces other experts to take part. The slide cites Switch Transformers (Fedus et al., 2021). A small inset video shows a lecturer at a board.
u120 37:37 a load balancing auxiliary loss, which is aimed at forcing your model to also incentivize exploring other experts by encouraging the fraction of the tokens that are being rerouted to be uniform or more or less uniform
u121 38:00 across experts. And so maybe it's something that I didn't say, but I will say it now. The routing is done at the token level. So if you have an input sentence, each token will ask itself, which expert should I go to? So this is done at the token level.
u122 38:21 And so here this loss is incentivizing your model-- yeah. So the question is, well, if you select one or two, it's not differentiable. It's a great point. So you have an operator that's called stop grad that is treating some aspects as being actually derivative of 0
u123 38:42 and treating these as constants. So long story short is in the loss, you're treating the fraction of tokens routed to expert i as a constant. And you're only backpropagating through the routing probability. So you're still able to actually change those routers.
u124 39:02 But yeah, that's a good point. So let me see. So I think we're talking about the fact that these auxiliary loss is incentivizing your model to also choose other experts and not to have this routing collapse problem that we
u125 39:24 mentioned at the beginning. There are some other problems as well. I just want to just tell you that training an MoE based LLM is challenging, and this is part of its challenge. But the good thing is, if you do it properly, you will have-- so here on the slide, what you see
39:38 · Interpreting experts: colored MoE layer code from Mixtral · A Stanford ICME lecture slide titled "Interpreting experts" shows a PyTorch MoeLayer class with each token highlighted in a color. The slide says each color represents an expert, so the highlighting shows which expert handles which code tokens in Layer 0. A small video inset at top right shows a person standing in front of a board.
u126 39:46 is an input text where the color of each section denotes a specific experts that a given token was sent to. And this is, for instance, at a given layer. And this is coming from a paper from Mixtral. So here is the model called Mixtral of experts.
u127 40:07 And you see that the colors are pretty diverse. So it means that you're actually leveraging your experts pretty well. And so I want to just say a few things about MoE-based LLMs. So now we're in 2026.
u128 40:23 This is something that I think the paper was from 2024, which is quite some time ago in the AI field. So nowadays, what people do is they use experts, but they also have shared experts that gets activated no matter which experts the router
u129 40:46 or the gate selects. And the idea behind this is for every token, there is always some computation that is always necessary. And you want to avoid your experts from always learning the same thing as a foundation and then specializing into something.
u130 41:06 So that is something that people do nowadays. The second thing that people do is the number of experts, they don't choose it to be 8 or 16. They actually choose it to be higher, 60 or in the hundreds. And the reason why they do that is that intuitively, you just give the chance to your experts
u131 41:29 to be very specialized, very fine-grained. But also on the charts, you see that the performance, when you have more fine-grained experts is just higher. So it's also just empirical. And so speaking of order of magnitudes. So when it comes to the frontier open source LLMs
u132 41:51 that were published recently. So here the screenshot is actually from a few days ago. If we take the top models, what we see is that-- so there are MoE-based LLMs. And the number of experts that they have is on the order of hundreds. So here are 384, 256.
u133 42:12 And the number of experts that are activated at each forward pass is typically eight or so. And you also have a shared expert, which is not widely used. Like in this example, the first model is not using it. And in terms of the model size, it
u134 42:32 is in the hundreds of billions of parameters or even trillion parameter for the first one. So it's pretty big models. Yes. So it's a great question. The question is, does the scaling law still hold true for a MoE-based LLMs?
u135 42:51 So the answer is that paper was actually based on a dense LLM. It was not like an MoE-based LLM. So one thing that I've seen papers do is have these scaling law experiments for their own setup on small number of tokens, small model size to just see what the law is for the specific setup.
u136 43:15 So to your question, I think for MoE-based LLMs, if authors feel that their choice, their design is very different from what was tested before, they would just do this very small experiment and see what is the kind of law that comes from there,
u137 43:35 and then choose the parameters from there. Yep. Yep. So the question is, why is n the number of experts doing in the numerator? So here, actually, what you're doing is you are summing across the experts.
u138 43:53 And so if you actually write that mathematically, you actually want to add n to-- because fi and Pi is like over n, so it's over n squared, times n is like over n. So it's times n. But n is the number of experts.
u139 44:13 OK, great. We're doing amazing. Any questions, other questions? So far, what did we see? We saw that people have tried different variants of the transformer.
u140 44:31 They saw OK, how about we take the encoder? Well, OK, you can do this. How about you take the decoder? You treat that as a text-to-text problem. They scale that up. And people have seen that the performance just gets better and better as people increase it in size and in number of tokens that it is trained on.
u141 44:50 We've seen that when the model is too big, you need to do something in order for you to not have crazy inference costs. And so that's why people have developed a MoE-based LLMs that allow you to scale the number of parameters without having to incur the cost of also activating
u142 45:11 all the parameters in each forward pass. So here in this part, what we will focus on is a number of changes that people have made in the layers that were originally described in the original transformer architecture.
u143 45:33 So you will see in a second what I mean.
u144 45:36 Position embeddings
u145 45:36 So if you remember, the transformer architecture relies on the self-attention mechanism, which is the idea of letting all your tokens communicate with everyone. So the problem when you do that is
45:36 · Need for position information in attention · A Stanford lecture slide on why transformers need position information: direct query-key links between tokens "lose" position info. It shows the sentence "a cute teddy bear is reading ." with a value vector above and a transposed key vector below each token. The query for "teddy bear" connects by lines to every key, including its own.
u146 45:56 that you lose the information of where tokens are positioned because you have these direct links. So if you don't have any information about positions about each of these tokens that are included anywhere, well, you lose the information of where tokens are located. So the original transformer paper tried two approaches.
u147 46:23 The first was learning position embeddings and adding those embeddings to the token embeddings. So what people would do is you have an input sequence, you have your tokens. Each token is associated with a position index.
u148 46:42 Let's assume 1, 2, 3, and so on. And here the idea is to learn an embedding that is associated with each position index. So position 1 will have an embedding, position 2 will have an embedding, and so on. And these embeddings would be learned.
u149 47:02 And what would happen is, in order for your model to actually know about where each of these tokens are located, so if you see on the right, what people would do is take the token embedding and then add that position-specific embedding to the token
u150 47:23 just to reflect that token is at a very specific position. But the problem with this approach is, as we mentioned last time, that if you train your model on a given length of input data, if you ever come across some sequence at inference time, that is,
u151 47:48 let's say longer, the problem is actually have never encountered positions beyond the ones that you encountered at training time. So you actually do not know what embeddings for such position are. You don't know.
u152 48:06 And so that's the reason why the original authors of the transformer paper, they not only tried the learned way, but they also tried a deterministic formula-based way that we're going to see in a second. And the reason why they tried this approach
u153 48:23 is because it can generalize to any sequence length without having you to actually train on longer sequences. And they actually said that both approaches were roughly leading to the same performance.
u154 48:42 So the fixed one is better and we're going to see it now. So there was this analogy that I mentioned last time that I will start from in this session. So the idea behind the deterministic formulation is to express each position on a high dimensional vector.
u155 49:11 And what you want is that vector to exactly inform you in which position you're in. And so you want that. You also want two positions to be embedded, such that the similarity operation between these two
u156 49:35 embeddings are high if the positions are close and it's low when these positions are far. And so I'm going to just motivate how the authors have actually designed the formula and the design that they're
u157 49:52 using for the position embedding that they're proposing here. So going through the analogy that we discussed, let's assume you're looking at your watch. You have several hands. You have the hour hand. You have the minute hand.
50:05 · Hardcoded position embeddings: clock analogy, hour hand · A Stanford lecture slide titled "Hardcoded position embeddings" states the idea of hardcoding position-embedding values. It uses a clock face with a single blue hand, labeled "Hour (slow)", as an analogy for encoding position with a cyclic signal. A small inset in the top right shows the lecturer standing in the room.
u158 50:09 You have the second hand. If you put all these hands together, you know exactly what time it is. And each of these hands, they're moving at different frequencies. So the hour tells you roughly when you are in the day,
u159 50:28 the minute is a little bit more precise. And then the second is like, even more precise. And so what we're going to do is do something very similar for the position embeddings, where we'll have a function of higher and lower frequency, such that when you put all of that together into one vector,
u160 50:54 if you know exactly which position you're at. So it's going to get a little bit more into the math. So disclaimer, I hope you remember your trigonometry formula. But if not, don't worry. We'll also look at it together. So let's assume that you are at position m.
u161 51:16 And so here you're looking at two consecutive indices-- so 2i and 2i plus 1. So what the authors propose is to have one be a cosine, the other one be a sine or vice versa,
u162 51:35 and have the frequency of these functions be a function of the index. So here, if i is low, then you have a higher frequency function. If i is high, you have a lower frequency function,
u163 51:55 meaning your sine will vary more slowly. So here, if you see on the graph, I'm not sure if you exactly read the axis. So on the right side of the slide, you have on the y-axis the position, and on the x-axis, the dimension.
u164 52:18 So here what you see is if you take the first row and you're moving down, you're going from position 1 to later positions of your sequence. And you see that the first dimensions, the first indices,
u165 52:37 the one on the left, they tend to vary very quickly, and the one on the right, the later indices, they tend to vary much, much more slowly. So in other words, your lower indices, they're much higher frequency. And the later indices, they are of much lower frequency.
u166 52:58 So just to tell you that this seemingly scary formula is just telling you this. So it's varying cosine, sine with different frequencies. It's a higher frequency for a lower index, lower frequency for a higher index. So let's assume you note omega i, the huge formula,
53:13 · Hardcoded sinusoidal position embeddings from "Attention Is All You Need" · A Stanford lecture slide explains hardcoding position embedding values. Even dimensions 2i get a sine of ω_i × m, and odd dimensions 2i+1 get a cosine of the same argument. Arrows from the two formula boxes point to the matching cells in a vertical vector for position m. A small inset shows the lecturer standing at a board.
u167 53:19 so that we don't need to worry about it. Well, what happens if you have a position m and a position n? How do you compute the similarity? Well, you take the dot product. So if you remember the way you take the dot product of two vectors is you multiply each dimension and then you add everything up.
u168 53:43 So just a refresher on trigonometry. So cosine of a minus b is cosine of a cosine of b plus sine of a sine of b just the trigonometry formula. So here if you take the cosine of omega i of m minus n,
53:58 · Hardcoded position embeddings using the cosine difference identity · A Stanford ICME lecture slide titled "Hardcoded position embeddings" with the idea of hardcoding position-embedding values. It faintly shows the identity cos(a−b)=cos(a)cos(b)+sin(a)sin(b), then applies it to cos(ω_i(m−n)), so a relative position m−n is written in terms of sines and cosines of the absolute positions m and n. A small inset at top right shows the lecturer standing in a lecture room.
u169 54:06 it's actually the dot product between these two indices. So the question, is i a function of the dimension? Yes.
u170 54:22 So i is a function of the index. So if you have a lower i, then your omega i will be such that it will be of higher frequency. So yes, it's a function of the index. That's a great question. So here going back to our favorite trigonometric formula, so here I just want to point out that cosine of omega i m minus n
u171 54:49 is equal to the dot product of these two. And if you actually want to take the dot product of the whole two vectors, you actually obtain a sum of the cosine of the omega i of all the indices multiplied by m minus n.
54:55 · Hardcoded position embeddings using cosine identity · A Stanford ICME lecture slide on hardcoding position embedding values. Faded trigonometric identities (cos(a−b) expanded into cos·cos + sin·sin) lead to the main result: the inner product of position embeddings PE_m and PE_n contains terms like cos(ω_i(m−n)), so it depends on the relative offset m−n. A small video inset shows the lecturer standing at a board.
u172 55:16 Is everyone following so far? Yeah. So why am I saying this? I'm saying this because the dot product of the two position embeddings at position m and n is a function of m minus n. And you may remember that cosine of 0
u173 55:38 is the maximum value that cosine can take. So in other words, when m is equal to n, the sum is maximized. So the similarity is maximum. So all of that to say that if you take this expression that is graphically something that you see on the left in terms
55:50 · Hardcoded position embeddings in transformers (Stanford lecture slide) · A lecture slide presents the idea of hardcoding position-embedding values. The left heatmap shows sinusoidal embedding values for positions 0–50 across about 128 dimensions, with fast oscillation at low dimensions and slower, then near-constant stripes at higher ones. The right heatmap shows similarity between positions, with a strong dark diagonal that fades as positions move apart. A small video inset shows a person standing at the right.
u174 56:00 of the value of the embeddings, what you will obtain is that positions that are close together, they're actually of high similarity and positions that are farther together, they are of lower similarity. And that's what you see on the right. So on the right, you have the positions on the x-axis
u175 56:21 and on the y-axis of m and n. And you see that when the positions are equal, the similarity is maximum. So all of that to say that if you were to look at the formulas without any context, I think you'd be probably scared.
u176 56:39 But I hope that with this perspective, it's a little bit less scary. So now what I want to tell you is that if everything was great, we would be using this exact method today.
u177 56:59 But the truth is we are not. This is actually not something that people use today. People use something differently. And the reason why that is the case is if you think about it, you care about where tokens are positioned in a very specific layer.
u178 57:21 You only care about it in the attention layer, because the attention layer is the layer where you actually are communicating. You're trying to determine which tokens are relevant when you do your expression. And so this is the reason why people have actually
57:38 · From absolute to relative position info in attention · A Stanford ICME lecture slide on moving from absolute to relative position information. The motivation is that relative position matters in the attention layer. The slide proposes changing the attention layer instead, with a faded example of query/key/value vectors for "a cute teddy bear is reading." and a faded Transformer diagram from "Attention Is All You Need." A small inset shows the lecturer standing at a board.
u179 57:43 tried to find a method that is not affecting all the layers, but that is being surgically applied to the attention layer. Yeah. So the question is if you do that, then you may make
u180 58:03 further token interact less. And that was maybe the problem of RNN. So why are we doing this? So here the idea is to inject some position information and intuitively tokens that are farther apart, they're just less relevant.
u181 58:21 But they're not relevant. And so here the idea is that on a relative sense, closer token are just more likely to be more relevant. But you're still keeping the direct connections, and you're still giving a chance to your model
u182 58:39 to actually interact with those tokens. So I do get your concern. That being said, this is something that proved to work well, and it just reflects the idea that if let's say you have a 10 document page, then probably what you have in the same sentence
u183 58:59 is probably more relevant than something that is very far apart, but you're still giving a chance to these tokens to interact. OK, great. So going back to here. So one major drawback of the previous approach is that this position embedding was actually
u184 59:20 being added to the token embedding, which was then being processed by all the layers. But intuitively, this is actually not what you want, because what you want is to let your attention layer know when two tokens are far apart. But you don't want to let your feedforward neural network know
u185 59:43 that your token is at position 5 or at position 12 because the feedforward neural network doesn't care. Can very well have a sentence where a token is at position 24 versus 1. That can be very making sense just because you
u186 1:00:01 have multiple sentences. So this is the reason why people have tried to mitigate this drawback by surgically looking into the attention layers and doing something there. And in particular, I'll go through a few methods
1:00:18 · Linear bias in attention layer (T5 bias) · A Stanford lecture slide explaining how a bias is added to query-key attention scores. A diagram shows the query for "teddy bear" (position m) attending to the key for "cute" (position n) in the sentence "a cute teddy bear is reading .". The softmax formula adds bias(m,n) to the scaled dot product, and the T5 bias is learned per head as a function of bucketed relative distance m−n.
u187 1:00:22 that people have tried. So one of them is a method which was meant to learn a bias term within the Softmax. So if you remember in the attention layer, you have this formula which I recommend you remember, which is Softmax of qk transpose over some scaling constant times
u188 1:00:45 v. So within the Softmax, you're actually trying to ask yourself how relevant the query is to all the other keys. So this is where that matters. And so here the idea is to learn a bias that is dependent on how far your token m and your token n
u189 1:01:11 are by learning a bias. The problem with this is, so similar to the very first method, you need to learn the bias. And so first of all, maybe your training data has some bias in itself. So it will maybe not optimal.
u190 1:01:29 But then the second thing is it's not generalizable to all possible distances. So that's why people have tried other methods as well. Another one is called ALiBi-- so Attention with Linear Biases. And so here the idea is to actually make that bias vary as a function of m--
u191 1:01:52 sorry, n minus m. So the distance between m and n.
u192 1:01:58 Rotary position embeddings (RoPE)
u193 1:01:58 But the method that is commonly used nowadays is not ALiBi, is not the T5 bias, it's RoPE. So who has heard of RoPE? Yeah? OK, great. So we're going to see why RoPE is the way it is. So RoPE stands for Rotary Position Embedding.
1:01:58 · Linear bias in attention layer (ALiBi) lecture slide · A Stanford ICME lecture slide explaining adding a bias to query-key attention scores. It illustrates with the sentence "a cute teddy bear is reading." where query m ("teddy bear") attends to key n ("cute"). The softmax formula adds bias(m,n), and ALiBi defines it as a linear, deterministic, unbounded function of distance, while the T5 bias is greyed out.
u194 1:02:21 And the idea behind RoPE is in the attention layer, you're not going to add a bias. What you're going to do is you're going to change the query and the key directly. And so how are we going to change them directly? What you're going to do is you're
u195 1:02:40 going to rotate your query. You're going to rotate your key, each of them as a function of the position that they're in. And why is it something that is very appealing? Well, because the nice thing is when you rotate the query, when
u196 1:03:02 you rotate the key, what you're actually doing, when you do the self-attention formula is you're actually bubbling up an expression that is a function of the difference between n and m.
u197 1:03:25 So we're going to see a little bit how that works. So here, so I'm not sure if any of you is an expert at geometry and rotation matrices. But the way you rotate a vector is you take that vector and you actually have a rotation matrix
u198 1:03:45 that you apply to that vector. So I'm going to maybe just write that down. If I do have this. So let's suppose you have some vector, let's call it x. So you have, let's say you have two dimensions--
1:03:57 · Lecturer at blank chalkboard in Stanford classroom · A still from a lecture video shows a person with short dark hair, glasses, a dark T-shirt and green trousers standing with their back to the camera. They raise a hand toward the upper left of a large, empty dark-gray chalkboard, as if about to write. A faint "Stanford" watermark is at the lower right, and the edge of a projected slide is partly visible at the top right.
u199 1:04:08 let's assume you have x and y. And let's assume that your vector is like this. And this is let's say a radius of r. So here the x-axis is x-- sorry, it's R and here you have your theta angle,
u200 1:04:32 cosine of theta. And then here you have your y value, which is R sine theta. So let's assume you have this. So what I'm only doing is expressing
u201 1:04:51 x as a function of some angle theta that it already has. And what I want to show you is if you were to multiply the rotation matrix with this vector x. So let's assume you have a rotation matrix of let's say angle phi, let's assume, which is-- so I'm going to just copy
u202 1:05:14 what is on the slides. So it's cosine of phi is minus sine of phi, sine of phi, and then cosine of phi. So let's assume you actually multiply your rotation matrix
u203 1:05:37 with the vector x. So for those of you who love matrix multiplications and trigonometry formulas, this is the best case because I'm going to exactly see. So here you would do the matrix multiplication.
u204 1:05:57 So R times cosine of theta cosine of phi minus R cosine of theta-- sorry, minus R sine of phi sine of theta. What is this? It's R cosine of theta plus phi. And you guessed it, the other one is the same, but with sine.
u205 1:06:30 So why am I writing all of this? What I want to show you is if you multiply your vector with the rotation matrix, you're rotating your vector. And this is the reason why we have this formula written on the slide. And the nice thing is, if you rotate your query vector
u206 1:06:52 by some angle as a function of m, and if you rotate your key vector as a function of its position n, then you obtain a rotation matrix that is actually a function of n minus m.
u207 1:07:11 So I will spare you the details. But if you just do the multiplication, you will see that it works. And so the reason why it is pretty nice is because this is extensible to any positions. You're not learning anything.
u208 1:07:30 And by the way, one other thing I will say is I just showed you the 2D case. But in order to generalize that let's say d-dimensional case, you would actually do that by 2D blocks. And you would do the rotation with angles that
u209 1:07:54 are a function of the index. So to your earlier question, so here you would have also angles that would be a function, let's say if it's a lower index, it's a different angle compared to if it's a later index. Cool.
u210 1:08:12 And the nice thing with this method is if you take a look at the order of magnitude of the dot product between the query and the key as a function of how far their positions are, then you will see that roughly it tends to decay as m and n
u211 1:08:33 are farther apart. So it's not strictly monotonic because, well, it's more complicated than that. But at a very high level, it just decays and it just reflects what we want in tokens that are very far apart.
u212 1:08:53 So that was RoPE. So in other words, the original transformer paper used position embeddings that were additive, which is in contrast with nowadays where people use RoPE as a way to encode position.
u213 1:09:13 So this is how people use it. And they don't use the additive position that we discussed before. So now we're going to talk about another component that is a little bit different compared
1:09:28 · Sparse attention: Longformer slide with empty attention grid · A Stanford lecture slide titled "Sparse attention: Longformer" shows an 8x8 grid of pink cells meant as an attention matrix. Rows and columns are labeled with the tokens [CLS], A, cute, teddy bear, followed by ellipses. No cells are highlighted yet, so the sparse pattern has not been revealed. A small inset shows the lecturer standing at the top right.
u214 1:09:28 Sliding Window Attention (SWA)
u215 1:09:29 to the original transformer, and this one is the attention layer. So I mentioned that in the encoder part, everyone can attend to everyone. In the decoder part, a given token can attend to itself and to tokens before it. So one natural question that people have asked
u216 1:09:52 is, well, do you really need to attend to everyone at every layer? Because it's a lot of computations. And the answer is, well, maybe you don't need to do that. And in particular, there is a specific variation
u217 1:10:14 that is called the sliding window attention that is actually meant for you to attend to only up to a certain position that is farther away from you. Not everyone, but only up to a certain level. And so here it's something that people also
u218 1:10:34 call local attention. And what I want to tell you is you may-- so probably, you will probably come back to me with saying, OK, the RNN actually had this issue. Well, why are you doing this? So I want to point out something that I hope will be making sense.
u219 1:10:59 So your model is not just one block. It's not just one block of the decoder. It's actually several blocks. So the attention operation is not just happening once. It is happening several times. So if you have, let's say, a sequence of tokens.
1:11:17 · Lecturer at blank blackboard, Stanford classroom · A man seen from behind, in a dark T-shirt and green-gray trousers, raises his hand with chalk to a large empty dark blackboard. A faint "Stanford" watermark is at the lower right, and part of a projected slide is at the top right edge. The board is still blank, so he appears to be about to start writing.
u220 1:11:25 So I'm going to just replicate exactly these tokens over here and let's say over here. So let's assume that this token can only interact with itself and with one token from before.
u221 1:11:54 Here you only see yourself and the token before. But if you consider that in a similar fashion this token can attend to itself and the token before, well, effectively, this token is able to see representations
u222 1:12:15 from tokens that are beyond the limit. Does what I'm saying making sense? Yeah. OK. So in other words, even though you are limiting yourself in the tokens that you're able to interact with,
u223 1:12:35 if you have several stacked blocks, then what you're actually able to do is to interact with further tokens. And for those of us who are also familiar with the vision world, so in case you're familiar with convolutional neural networks, it's a little bit the same idea as receptive fields.
u224 1:12:58 So here when you perform the convolution, a given element is actually a function of actually many pixels if you go back into the chain. So it's the same idea. OK, I want to tell you something else about the attention layer.
u225 1:13:15 Group Query Attention (GQA)
u226 1:13:18 So I think we briefly discussed it earlier in the lecture. But OK, I'm going to reuse this. So let's assume that we're back with our decoder only model which is typically what LLMs are composed of.
u227 1:13:41 And let's assume that what you want to do is to predict the next token. Well, this is your last predicted token. In order to predict the next token, you have a bunch of layers, but in particular in the attention layer, what you're doing is you're attending to yourself.
u228 1:14:01 You're attending to the token before, the token before that, the token before that. Let's assume you predict a new token and that the new token, in order to predict the next token, you're attending to yourself, you're attending to the one before you, and so on and so forth. So every time you want to predict the next token, what
u229 1:14:27 you're doing is this causal attention, where you're comparing the query from your token with all the keys from the other tokens. And every time you want to predict a new token,
u230 1:14:45 you again compare it to the keys of the previous tokens. So you could either compute all the previous keys every time you do that operation, or you could store the keys as you perform your decoding.
u231 1:15:09 So, for instance, when I decode when I'm here, I attend only to myself. I computed the key. You can save the key. You predict this one. So here I attend to here and to the one before-- so I need the key. I can reuse the key. And I compute the key here that I save. And I'm continuing this process so on and so forth.
u232 1:15:29 So the problem with this is keys detect memory space. And as your decoding gets longer and longer, you need to store these keys. And so if you remember, the attention layer is actually a multi-head attention layer.
u233 1:15:48 So it doesn't only have one key, it has actually several keys for the same layer for the same token. So that is the reason why people have tried saving up the number of keys and the number of values that each token needs to produce. And that's why you have this method.
u234 1:16:07 So, for instance, the idea of sharing keys and values across heads. So the reason why people are interested in such methods is because given that you always need the keys in order to predict the next token, you want to save memory in there.
u235 1:16:30 And so that is the reason why you have a number of methods like group query attention, which shares the key and the value projection matrices across the heads, or the group query attention which is a little bit more common, where you have a certain number of groups where you share the key
u236 1:16:51 and value matrices. And this contrasts with the original multi-head attention layer, which had one heads, where you had all the projection matrices that you had to learn. So all of that to say that given that you
u237 1:17:11 are in this situation where you're always saving your keys and your values for tokens that you have decoded in the past, what you want is to not have to save too much of them. You don't want to saturate your memory.
u238 1:17:29 So people are trying to find ways to optimize that. And one such way is to just reduce the number of different projection matrices that you need to learn for the keys and the values. And that is the reason why, I mean, no one cares about the query because the query,
u239 1:17:47 you use it once and then you don't need that. So does that make sense? So nowadays people use GQA quite a lot. So among everything we've seen, you will see a RoPE quite a lot. You will see local attention with sliding window quite a lot.
u240 1:18:06 You will see GQA quite a lot. And there is one last thing that I want to talk to you about,
u241 1:18:13 Pre-Norm versus Post-Norm
u242 1:18:13 which is the normalization. So in the original transformer architecture. So if you remember, so it's a little bit more of a detail when you first read the paper. But the way people do normalization is they first perform the sublayer operation and then they add to the inputs
1:18:13 · Post-Norm layer normalization in Transformers (lecture slide) · A Stanford lecture slide titled "Different kinds of layer normalization" presents the Post-Norm variant, where Output = LayerNorm(x + SubLayer(x)). A diagram shows a transformer block in which Multi-Head Attention and then an FFN each feed an addition (residual) step, followed by a Layer Norm, so normalization comes after the residual connection. A small inset shows the lecturer standing at a blackboard.
u243 1:18:40 and then they normalize. So they do normalization of x plus sublayer of x. So it's also called post-norm. So you're normalizing after the operation. And the reason why people thought it was maybe not the best way is because here
u244 1:18:58 the idea behind residual connection is that you want to let the input through just for the gradient to flow better. And so here the x is actually within the layer normalization. So it's not great. And so this is the reason why people nowadays, what they do is they don't normalize after the operation.
u245 1:19:19 They actually normalize before. So they first normalize the inputs and then they put it into the layer. And then they add it to x. It's also called pre-norm. And it's just a way for the residual connection to do its job of letting the gradient flow. So you will see pre-norm as also a very common method.
u246 1:19:41 LayerNorm versus RMSNorm
u247 1:19:42 And one last thing, I promise, which is the layer normalization itself. So in the original transformer, here we're normalizing across all the activation by first recentering x and then having a term, gamma, and then
u248 1:20:02 some bias beta, which helps with training stability and convergence. But people have tried with other normalization methods. And in particular, they have found that RMS norm-- RMS stands for Root Mean Squared norm, actually does the job just as well.
u249 1:20:22 And here, you don't need to recenter x, you just need to rescale it and to have just one factor that you need to learn. And that is the reason why people tend to use it. And I think this was all the tricks I had. So just as a summary.
u250 1:20:41 Before on normalization front, people used to do post-norm and layer norm, and nowadays people do pre-norm and RMS norm. So long story short, nowadays RoPE, local attention, gka GQA, pre-norm, and RMS norm.
u251 1:21:07 And with that, I'll give it to Shervin.
u252 1:21:09 Next token prediction
u253 1:21:11 Thank you Afshin. So now we're going to spend some time together to talk about next token prediction. And as Afshin has mentioned, LLMs they use this autoregressive process where you start with a given set of tokens that have been processed, and then you generate the others
u254 1:21:31 one at a time. So let's say you start with just BOS. So you generate A and then teddy, bear, is, and then you complete the sequence. So now the question is, how do you generate the next token? So you know that all of these are processed
u255 1:21:53 through these decoder layers. So you have the self-attention mechanism and the feedforward neural network that is repeated n times because you have n layers of decoder that are stacked. But then at the end, it is projected on a linear layer to predict the next token on the space of your vocabulary.
u256 1:22:16 And there might be several ways to choose it. So what could be a natural way? So first, since you have a probability distribution, one piece of intuition could be the probability distribution has done a lot of work for me.
u257 1:22:34 It says, what is the most probable token? So let's just take it. So that could be one. But the thing is, as you compute tokens one after the other, you will try to optimize the next token immediately. But as you have generated the whole sequence,
u258 1:22:56 you see that the global sequence might not be with optimal joint probability. So this is one weakness of this approach. And you might be tempted to go the joint probability route a bit more. So this is why you have techniques like beam search
u259 1:23:17 that doesn't do full joint probability because that would be too computationally expensive. But it does track k path at a time. So let's say you want to compute the next token, instead of only selecting the next one, you will consider k path all the time.
u260 1:23:36 So here you have beginning of sequence. You have an there that are good like the two best ones. And then you generate the next probability distribution, and then you keep on keeping the k best and so on. So who can tell me, what kind of weakness
u261 1:23:58 you see out of these two techniques that I haven't called out just yet? Yes. So there is a more computation. Yes, for the second one, but not for the first one. I'm looking for something that is common to both techniques. So what can you say about the reproducibility
u262 1:24:25 of choosing always the highest probability token? It will always be the same. What can you say about beam search? Let's say you redo the computations at the same time. Do you get the same path each time or will it change? Yeah, great.
u263 1:24:46 So yeah. Same path and it's deterministic. Great. And this is where I want to bring your intuition on why that may not be great, because as someone who optimizes things for a living, you might say, OK, I don't mind.
1:24:49 · Predicting next token with beam search · A Stanford ICME lecture slide explaining beam search as a second idea for decoding: keep the k most likely paths. A tree starting at [BOS] highlights the path a → cute → teddy/bear, with other branches faded. The slide notes the problem: more computation and still lacking creativity. A small inset shows the lecturer, and a transformer diagram sits on the left.
u264 1:25:05 Determinism might be good. But I just want to show you as a user why you would care about it. So let's suppose you are a chatbot user of all our favorite LLMs, and you ask a given question, like let's say a greeting message, "Good, and you?" And let's say your friends ask the same message
u265 1:25:29 and you have the same answer, and so on, you might feel that the chatbot is robotic. And robotic chatbots, I don't know about you, but it's not so appealing because it feels like it's still heavily engineered, whereas you as a human, you might be tempted to respond a bit differently
u266 1:25:49 to similar prompts. So what I said here is mostly a thought experiment, because in practice, the context of the LLM might change quite a bit. Let's say today we are the 2nd of October at Stanford. If I ask this question one hour from now, the context will change a bit.
u267 1:26:08 It might put my new time and other pieces of information. So this might not strictly apply. But still you see this robotic behavior. Whereas non-deterministic behavior could give similarly well-sounding answers as potential choices, which is great.
u268 1:26:30 So I hope I convinced you that determinism is not the best. So it can be good for some use cases. I'll talk about it in a bit. And one natural next step to combat determinism is to go back to this probability distribution
1:26:41 · Predicting next token by sampling from probability distribution · A Stanford ICME lecture slide presenting the "3rd idea" for next-token prediction: sample the next token from the model's probability distribution P(w_{t+1}|C). A bar chart gives probabilities over candidate words, with "kind", "smart", "fluffy" and "gentle" highest and "airplane" and "where" lowest. On the left is the transformer decoder diagram with the Output Probabilities layer highlighted, and a small video inset shows the lecturer at a blackboard.
u269 1:26:51 and simply sample from it, because you have a probability distribution, you have weights. All our computational methods enable us to sample the next word. So let's just do that. And you can do so with two methods, because you might not want to deal with the whole distribution
u270 1:27:11 all the time. So you might want to select amongst top-k most probable next tokens. And the second technique that is used quite often is top-p where you want to select tokens that have a given probability
u271 1:27:30 distribution in cumulate. So I promised you that I would tell you when determinism is needed. So, for example, if you perform natural language tasks that are tricky and need reproducibility,
u272 1:27:50 for example, let's say you want to summarize a piece of text. You want the same output all the time, that can be good. In one of the later lectures, we're going to talk about evaluations and particularly LLM as a judge, where you want consistency between the ratings that you give. So these two are good examples.
u273 1:28:12 In the rest of the time, this non-deterministic path might be the most attractive. So now I'm going to go into the specifics that I mentioned on a high-level, but making sure that it makes sense. So in the last layers, you have this linear projection
u274 1:28:31 between d model, the embedding space of your token at a given time. And then you project it on the space of vocabulary. And then based on this, you apply this Softmax operator to convert these numbers into a probability distribution.
u275 1:28:52 And why am I telling you that? It's because I want to introduce a parameter that you might have heard as users quite bit-- the temperature T, which helps modulate the probability distribution that you get. So here, as you see, so the regular Softmax operator
u276 1:29:16 has T equals 1. So the formula that you see on the right is T equals 1. You introduce T like so. I want to ask your intuition of how you feel the probability distribution to be then. So let's say you take T bigger than 1.
u277 1:29:35 If you have a given probability distribution, how do you feel the probability distribution will change? So yeah, the one comment is flatter. Yeah, great. And let me ask you the opposite. Let's say you have very small t.
u278 1:29:53 What is your intuition? Yeah and the comment is sharper. Yeah, great. So is it clear to everyone? Because if you have small T, then the Xi that is high will be more emphasized. And the for the other i's, the some at the bottom
u279 1:30:14 will take it over. So you will have just one term-- so you will have just one index that will have a probability close to one and the other ones will be close to 0. And when T is large, then all of these x exponential of something over T will tend to 0 within the exponential. So everything will be more or less uniform.
u280 1:30:37 And this is exactly what these graphs show. And one thing, so this is a determinism is continuous more so than truly like binary choice. So you're not committed to either turn it on or off. You can adjust the temperature as a setting.
1:30:37 · Impact of temperature on probabilities (lecture slide) · A Stanford ICME lecture slide titled "Impact of temperature on probabilities". It lists "Small T" and "High T", each paired with a question mark, so the effects are left blank as questions for the audience. A small inset in the top right shows the lecturer standing in front of a blackboard.
u281 1:30:58 And I want to say-- so I talked about determinism. Does it really happen? So you could think that it does, because when you look at your decoder architecture, you have embeddings that are learned. OK, these are deterministically set at inference time. It goes through the masked self-attention mechanism.
u282 1:31:17 You have fixed weights, fixed operations. You could think that's fixed, feedforward neural network. You have given weights that are fixed. You multiply. This should be fixed and so on. So with a temperature of 0, you would think that this is completely deterministic. But it turns out that not because in practice LLM
u283 1:31:39 inference engines, they will batch your request among others. And the reason why nondeterminism comes into play is a purely computational one. In theory, it should still be deterministic. But there is this concept of batch invariance that is not respected in some cases.
u284 1:32:00 And I strongly recommend this blog from [INAUDIBLE] as a great piece of reading. So he goes through the exact reason. And if you want me to break it down to you, it all goes to the floating points non-associativity with respect to the addition operation,
1:32:13 · Person drawing box-and-arc diagram on chalkboard · A person in a dark t-shirt and glasses, seen from behind and the side, draws with chalk on a dark chalkboard. On the board are four hand-drawn rectangles in a row, with many curved arcs looping over them and connecting them, and small tick marks beneath each box. A faint "Stanford" watermark appears at the lower right.
u285 1:32:22 where the order of addition does matter on the result that you get. So let's say you add a big number to a small number and then you retract that big number, you might get 0.
u286 1:32:42 Whereas if you do that operation separately, you will get that small number back. So it all has to do with the representation of floating point numbers that are approximated. So this relationship does apply in the world of floating points. Yeah.
u287 1:33:05 So the comment here the tensor should be computed independently. So it turns out that depending on the implementation, the order of operation within these tensors will change. Yes, that is the reason. So yeah, the blog has more information on exactly why
u288 1:33:28 and then how to build kernels that do apply this as great reading. Any other questions? OK, great. And so now we talked about decoding in the usual case. Sometimes, and more particularly in an industry setting,
u289 1:33:50 you might want to use these LLMs to perform tasks you would have used complex systems for. So you might want to generate let's say structured output, such as JSON. And one question that you might ask yourself, let's say you want to generate a piece of data
u290 1:34:10 in this specific format and then this matters to you. Because if it doesn't do it right, then parsing fails downstream and you're in trouble. How do you do so? So one first approach that you could think about is just ask the LLM for it, do this in JSON format and hope for the best.
u291 1:34:29 With today's models, even the lightest, this usually works well. But it's not 100% and what you want is 100%. So how can you do that? So there is a concept called guided decoding, where the span of available tokens to be predicted at the next step is restricted,
u292 1:34:52 depending on the syntax of the language you want to decode this in. So let's say you are in JSON. You know That your first token must be a specific symbol and so on. And this is how you could deal with it.
u293 1:35:14 OK, so I hope all made sense. And now we're going to talk about one exciting part, which is prompting.
u294 1:35:22 Prompting
u295 1:35:23 And I want to introduce some terms, so that we all know some basics of vocabulary there. So let's say you want to issue a request to an LLM. Usually, you do not put just the BOS token and let it generate everything afterwards.
u296 1:35:42 You have a given prompt. So here my prompt could be a teddy bear. And I want to generate the next tokens out of it. So what happens is that I will put all of them as input. And then autoregressively, it will compute each token one at a time.
u297 1:36:01 And you might have heard of this concept called context window. Where does it come at play here? So at the stage of next token prediction, anywhere you are, what is as input is your context window. And when you look at your whole prediction,
u298 1:36:23 once it's completely done, whatever tokens were generated are called the output tokens. So you might be tempted sometimes to say that the context window is exactly what you put as input as a user, but it actually is more than that, because every token that you generate
u299 1:36:41 needs to be put back at the input level, which is why when you think about these large numbers, you think I can input anything. These are large numbers as context window, as we'll see in the slide. You need to take that into account. So it's linked with output length.
u300 1:37:00 Yep. So the question is, why if you need to have a fixed context length, like a maximum context length? So the reason is a computational one. And we'll see very soon why it might not even be desirable to have higher context length. So there is also an intuition based one reason.
u301 1:37:21 We'll it in seen in a second. That makes sense? So I want to give some orders of magnitude here. So in terms of context length, you might all know about that O of 1 million tokens as input, which is the industry standard.
1:37:28 · Orders of magnitude: LLM API model specs compared · A Stanford ICME lecture slide titled "Orders of magnitude" shows API page screenshots for three models: OpenAI's GPT-6.1 Sol, Anthropic's Claude Opus 5.5 and Google's gemini-3.1-pro-preview. All three have context windows of about 1 million tokens, with output limits of 128K tokens for the first two and 65,536 for Gemini. Pricing is listed for the first two: $2/$10 per million tokens for GPT-6.1 Sol and $4/$20 for Claude Opus 5.5. A small inset shows the lecturer standing in a room.
u302 1:37:39 And then as output tokens, it's usually in the order of magnitude of 100k. That was true until two days ago where I had to rework this slide. Actually, maybe soon, we're going to move into the million territory there as well. And this new LLM that came out didn't publish an input maximum
u303 1:38:00 length. So maybe we might have gone one order of magnitude higher there. But these are some good orders of magnitude. And then to give you some intuition on what token means. So usually, a token is 3/4 of a word and a page is about 500 tokens.
u304 1:38:22 So when you talk about orders of magnitude, if you take 100k tokens, it's about a book, and a million, it's a large book. And now coming back to your question, is it the case that always having more tokens as input
1:38:38 · Token mental model and orders of magnitude, Stanford lecture slide · A Stanford ICME lecture slide titled "Discussion" gives rules of thumb for converting tokens to words and pages, then scales of token counts. Most of the text is faded, and only the closing question "Are more tokens always better?" is in full black. A small inset video shows the lecturer standing in front of a chalkboard.
u305 1:38:38 Context rot
u306 1:38:39 is better? Well, you need to beware of a phenomenon that's called "context rot," where there is this test called needle in a haystack, where you put a piece of information that you care about somewhere in that context window and the task is a retrieval one.
u307 1:38:56 And it turns out that as the context length goes higher, the LLM has more and more difficulty to retrieve that piece of information. So sometimes it's a problem that comes from the LLM itself. So you will always see in the model releases the hallucination rate that goes down and so on.
u308 1:39:18 But this is not something that you could perfectly solve, because conceptually, when you mix a piece of information, among others, it might be the case that there is so much noise around that the task of retrieving exactly that piece of information is not realistic. And in general, putting more and more as inputs
1:39:25 · Lecture slide on "context rot" with GPT-4 needle-in-haystack test · A presenter in a dark T-shirt and green pants stands gesturing in a Stanford lecture room with chalkboards behind him. A projected slide titled "Prompting: risk of 'context rot'" shows a heatmap from the Needle in a Haystack pressure test of GPT-4 128K. Retrieval is mostly accurate (green), but it degrades (red/yellow cells) at long context lengths, roughly 73K–128K tokens, when the fact sits at about 10%–50% document depth.
u309 1:39:41 pushes that risk up, which is why in general you hear about context compaction. You should start a new chat every time. These things still apply conceptually. And this is why we are not here with unlimited context window. It's also like a good practice.
u310 1:40:00 OK, awesome. And I'm going to spend the last minutes on techniques to improve performance without you changing any weights of your model. So there is a large concept called in-context learning, where instead of teaching in terms of weights tuning
1:40:16
u311 1:40:21 your LLM, you give examples and solutions of what you want. So let's say you are in the task of asking your LLM about the age of your teddy bear. So if you just ask it in zero shot versus give some question answers of what answer you expect,
u312 1:40:43 the LLM will learn the format and maybe even the reasoning that it needs to go through implicitly. And these days, you will see few shot learning being less and less used because the reasoning models, they get better and better.
u313 1:41:02 And the bad effects of few shot learning is that it will make your model overfit on the specific examples that you give it, whereas what you want is be more reasoning-based. So you will see some prompt tuning techniques that turn these examples into instructions, and people typically handcraft such instructions
u314 1:41:23 and put them instead of examples. And then again, some other obvious downsides are also you have more input tokens, which increases cost and latency. So on that reasoning part, you might say, how about we put some reasoning in there?
u315 1:41:42 Chain of Thought (CoT)
u316 1:41:42 So there is this landmark paper called chain of thought, where it takes in-context learning to the next stage by teaching the model in these few short prompts how it gets there.
u317 1:41:59 And the authors, they saw that the model learned the reasoning path, and overall it improved performance. And I don't want you to consider chain of thought to be purely an in-context learning technique. It's way more than that.
u318 1:42:18 And we will talk about reasoning models where thinking tokens are used and the same concept applies. So we'll talk about it then. And one technique that one might be tempted to use offline is called self-consistency. So how about, instead of generating one reasoning path, I
u319 1:42:38 generate several ones? Then one nice thing is that it makes the output that you might be interested in, let's say in the arithmetic world, more stable. And by majority voting, you can get to an answer where more generations agree to. Yeah, we'll see in reasoning models lectures, more
u320 1:43:00 about these. And that's all I had for you today. And I hope you have a great weekend. Thank you. [APPLAUSE]

The original

Around it

source

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processin

The Google AI Language team announces the open-source release of BERT, a pre-training technique for natural language processing. BERT is a deeply bidirectional model pre-trained on unannotated text, which can be fine-tuned quickly on small task datasets. The post reports state-of-the-art results on 11 NLP tasks.

Article

source

Improving language understanding with unsupervised learning

A single task-agnostic transformer, pre-trained on large unlabeled text with language modeling and then fine-tuned, reaches state-of-the-art results on many diverse language tasks. The largest gains come on commonsense reasoning and reading comprehension, which the author takes as evidence that unsupervised learning builds these skills. The work also carries high compute costs, limits from learning about the world through text, and brittle generalization.

Article

This text is written with AI.

Placed in: Now