Inside the forward pass

Priya owns the cooking assistant in Cookvale, a recipe app. She wants to see what a model actually is, so she downloads an open model, Qwen3-0.6B, from Hugging Face. The biggest file is model.safetensors, about 1.5 GB. Next to it sits config.json, 726 bytes. The rest are tokenizer files.

She also downloads its bigger sibling, Qwen3-1.7B, in a compressed version of about 1 GB. She runs it on her laptop and asks the same question twice: "Name one herb to add to tomato soup." The first time it answers "Thyme". The second time it answers "Dill".

Same file, same prompt, same laptop. Nothing in the file changed between the two runs.

So here is the puzzle for this lesson. What does the model compute when it reads a prompt? And where, exactly, does the choice between "Thyme" and "Dill" happen?

By the end of this lesson you will be able to work out a model's size from its config.json, follow one token through a pass, read a table of next-token probabilities, and explain how an API can promise that a tool call is valid JSON.

Two facts from earlier lessons carry this one. Lesson 1 showed that the model reads one long string of tokens and produces one new token per pass. Lesson 2 showed that a token is a piece of text stored as an integer, its token ID, and that the vocabulary is the full list of tokens a tokenizer knows. This lesson opens the pass itself.

What Is in a Model File

A model is a file of numbers. Each number is a parameter. Together, all the parameters are the model's weights. Training is the process that set them, long before Priya downloaded the file. While the model runs, they never change.

The file does not say how the numbers are arranged. That is the job of config.json. Here are its main fields for Qwen3-0.6B, copied from the file on Hugging Face:

{
  "head_dim": 128,
  "hidden_size": 1024,
  "intermediate_size": 3072,
  "num_attention_heads": 16,
  "num_hidden_layers": 28,
  "num_key_value_heads": 8,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "vocab_size": 151936
}

Each field is a size:

• vocab_size: 151,936. The number of tokens the model can read and write.
• hidden_size: 1,024. Inside the model, every token is carried as a list of 1,024 numbers. A list of numbers like this is called a vector.
• num_hidden_layers: 28. A layer is one of the identical blocks a token passes through, one after another.
• num_attention_heads, num_key_value_heads, head_dim: how the attention step inside each layer is split up (next section).
• intermediate_size: 3,072. How wide the MLP step inside each layer is (next section).
• torch_dtype: bfloat16, usually written bf16: a 16-bit number format. Each parameter takes 2 bytes.
• tie_word_embeddings: true. One table is used at both the start and the end (this matters for the file size, below).

Llama 3.1 8B, the model this course uses for GPU arithmetic, has the same fields with bigger values in its config.json: hidden size 4,096, 32 layers, 32 attention heads, 8 key and value heads of 128, MLP width 14,336, a vocabulary of 128,256, and no shared table.

From these sizes alone you can count every parameter in the model. Before you do, predict one thing. Each layer has two main steps, attention and the MLP. Attention is the part named in the title of the paper that introduced this design. Which step holds more of the parameters?

...

Count them. Every weight table in the model is a grid of numbers, so its size is rows times columns. This is the count for Llama 3.1 8B, from research done for this course and checked against Hugging Face's own total:

embedding table   128,256 x 4,096                    =   525,336,576
 
per layer:
  attention
    query    4,096 x (32 x 128)                      =    16,777,216
    key      4,096 x ( 8 x 128)                      =     4,194,304
    value    4,096 x ( 8 x 128)                      =     4,194,304
    output   (32 x 128) x 4,096                      =    16,777,216
    attention subtotal                               =    41,943,040
  MLP: gate, up and down, 3 x 4,096 x 14,336         =   176,160,768
  two norms, 2 x 4,096                               =         8,192
  layer total                                        =   218,112,000
 
32 layers         32 x 218,112,000                   = 6,979,584,000
final norm                                           =         4,096
output table      4,096 x 128,256                    =   525,336,576
 
TOTAL                                                = 8,030,261,248

The answer to the prediction is in the middle of that block:

MLP share of one layer        176,160,768 / 218,112,000 = 0.81 = 81 percent
attention share of one layer   41,943,040 / 218,112,000 = 0.19 = 19 percent

You might expect attention to be most of the model. It is wrong. In Llama 3.1 8B the MLP holds 81 percent of every layer's parameters, about four times as many as attention. Attention matters for other reasons: it is the only step where tokens exchange information, and its cost grows with the length of the prompt. The right model: most of the weights sit in the MLPs; attention is small in parameters and large in importance.

So the model has three bands. An embedding table at the start, 525 million parameters. Thirty-two identical layers in the middle, 6.98 billion. An output table at the end, 525 million. The layers are 87 percent of the model.

Now the bytes. Each bf16 parameter is 2 bytes:

Llama 3.1 8B   8,030,261,248 parameters x 2 bytes = 16,060,522,496 bytes = 16.06 GB

A GB in this course is 1,000,000,000 bytes. You will meet 16.06 GB again: it is what a GPU must hold before it can answer anyone.

Why Priya's File Is 1.5 GB, Not 1.2

The same count for Qwen3-0.6B, shorter:

embedding table   151,936 x 1,024                    =   155,582,464
one layer
  attention (query 1,024 x 2,048, key and value
    1,024 x 1,024 each, output 2,048 x 1,024,
    plus two small norms of 128)                     =     6,291,712
  MLP, 3 x 1,024 x 3,072                             =     9,437,184
  two norms, 2 x 1,024                               =         2,048
  layer total                                        =    15,730,944
28 layers         28 x 15,730,944                    =   440,466,432
final norm                                           =         1,024
output table      the embedding table, used again    =             0
TOTAL                                                =   596,049,920
 
bytes in bf16     596,049,920 x 2                    = 1,192,099,840 = 1.19 GB

In this small model the MLP is 60 percent of each layer (9,437,184 / 15,730,944), still the largest part. The vocabulary table is 26 percent of the whole model, which is why small models use one table for both ends.

But the file on Hugging Face is 1,503,300,328 bytes, about 1.50 GB. Where do the extra 0.31 GB come from? The file stores the shared table twice, once under each name, so Hugging Face's own counter reports 751,632,384 parameters. The few bytes left over are the file's header, a short JSON list of every table's name, shape and position:

596,049,920 + 155,582,464 (the table, a second time)  =   751,632,384
751,632,384 x 2 bytes                                 = 1,503,264,768
file size                                             = 1,503,300,328
difference (the file's header)                         =        35,560 bytes

When the model loads, the two copies become one table in memory. So "how big is the model" has two honest answers here: 1.19 GB of unique weights, 1.50 GB on disk.

You can count the parameters yourself. This needs Python 3.12 with torch 2.14.1 and transformers 5.20.1; the first run downloads the model, about 1.5 GB:

import os
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype=torch.bfloat16)
 
total = sum(p.numel() for p in model.parameters())
print(f"parameters:    {total:,}")
print(f"bytes in bf16: {total * 2:,}")
 
path = hf_hub_download("Qwen/Qwen3-0.6B", "model.safetensors")
print(f"file on disk:  {os.path.getsize(path):,}")

from_pretrained reads config.json, builds the layers it describes, and fills them from model.safetensors. model.parameters() lists every weight table once, so the shared table counts once. p.numel() is the number of values in one table. hf_hub_download returns the path of the downloaded file, and os.path.getsize reads its size in bytes.

Run on an Apple M1 Max laptop:

parameters:    596,049,920
bytes in bf16: 1,192,099,840
file on disk:  1,503,300,328

The count matches the worked total exactly, and the file is the 1.50 GB from above, with the shared table stored twice.

A model file is a table of numbers whose shape the config describes; count rows times columns and you have the parameters, and times two the bytes.

One Token, One Pass

Now follow one token through Llama 3.1 8B: the last token of the herb prompt, after the chat template from lesson 1 has wrapped the question.

Step 1: Look Up a Row

The token arrives as an integer, its token ID. The model uses that integer as a row number in the embedding table. In Llama 3.1 8B the table has 128,256 rows, one per vocabulary token, and 4,096 numbers in each row. The row becomes the token's vector.

This step is a lookup, with no arithmetic in it. That matters when we count the work.

Step 2: Run Every Layer

The vector now goes through 32 layers, one after another. Each layer reads the vector and adds an update to it, so the vector is a running total of the original row plus every layer's update. Each layer has two steps.

Attention. This is where a token looks at the tokens before it. In plain words: each token compares itself with every earlier token and takes a weighted mix of what it finds. The weight tables from the count do this work:

1. From its vector, the token computes three shorter lists of numbers with the query, key and value tables from the count: a query (what it is looking for), a key (what it offers to later tokens) and a value (what it hands over if chosen).
2. The token compares its query with the key of itself and of every earlier token. Each comparison gives one number: how well the two match.
3. Those numbers are turned into weights that add up to 1. A strong match gets a big weight.
4. The token takes the value of every earlier token, multiplies each by its weight, and adds them up. That mix is its update. The output table turns the mix back into 4,096 numbers.

Attention runs in several heads side by side, each with its own query, key and value, so each can compare tokens in a different way. Llama 3.1 8B has 32 query heads of 128 numbers each (32 x 128 = 4,096). It has only 8 key and value heads: each group of 4 query heads shares one. Lesson 5 explains why that sharing matters so much.

Nobody writes these comparisons by hand; training set the tables that produce queries, keys and values. When the last token of the herb prompt runs attention, it can put large weights on "herb" and "tomato soup" and small ones on the template's marker tokens. That is how the whole prompt reaches the position that chooses the next token.

Two properties of attention matter for the rest of this course.

• It is the only step where tokens exchange information. Everything else in a layer works on each token alone.
• It only looks back. A token compares itself with itself and earlier tokens, never with later ones. Models built this way are called causal. The paper that introduced this architecture says it modifies attention "to prevent positions from attending to subsequent positions." So when a new token arrives, nothing about the earlier tokens changes, because they never looked at it. Lesson 5 builds on this fact.

Here is attention as the original paper writes it (Vaswani et al., "Attention Is All You Need", 2017, Equation 1):

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

You do not need this formula for anything in this course. The four steps above say the same thing in words.

The MLP. The second step works on each token alone. MLP stands for multi-layer perceptron; in this course it means a block of matrix multiplies. In Llama's MLP, the first table turns 4,096 input numbers into 14,336 output numbers, and each output number is a weighted sum of all 4,096 inputs:

one output number = input 1 x weight 1 + input 2 x weight 2 + ... + input 4,096 x weight 4,096
                  = 4,096 multiplies and 4,096 adds
the 4,096 weights are parameters, used once each
14,336 output numbers x 4,096 weights = 58,720,256 parameters in this table

Llama's MLP has three tables of this size, called gate, up and down. Gate and up each turn 4,096 numbers into 14,336, their results are combined, and down turns the 14,336 back into 4,096: the 176 million parameters per layer from the count. Because most of the weights are here, most of a pass's arithmetic is here too. (Each step also starts with a small norm, a rescaling with 4,096 parameters: the "two norms" in the count.)

Step 3: Score Every Token

After layer 32, one more norm, then the output table. It has one column of 4,096 weights for every token in the vocabulary. Each token's score is the weighted sum of the final vector with that token's column:

score for one vocabulary token = 4,096 multiplies and 4,096 adds
scores produced                = 128,256, one per vocabulary token

Papers call these scores logits; this course says "score". A high score marks a likely next token, a low score an unlikely one. The model does not produce a word. It produces 128,256 numbers (151,936 on the laptop model, one per token of Qwen3's vocabulary).

A Whole Prompt at Once

In the first pass, the one lesson 1 called prefill, all the prompt's tokens go through the layers together, and in attention each looks back at the tokens before it. Only the scores at the last position decide the next token. The chosen token is added to the end, and the next pass runs for that one new token. That is the forward pass: one run through every layer, from lookup to scores.

Apply it to Cookvale. Turn 1 is 12,060 tokens: 2,000 of tools, 6,000 of system prompt, 4,000 of recipe and a 60-token question. In prefill, the last token runs attention against all 12,059 before it, so the recipe, the house rules and the question all reach the position that chooses the first word of the answer. A token in the middle of the recipe sees the tools and the system prompt, but never the question, because the question comes after it.

One pass turns a token into a vector by lookup, sends it through every layer (attention mixes in earlier tokens, the MLP works on each token alone), and ends with one score for every token in the vocabulary.

Where the Choice Happens

The pass ends with scores. Something must still turn 151,936 scores into one token. That something is the sampler: ordinary code that runs after the model.

From Scores to Probabilities

First the scores become probabilities, with a function called softmax. Every probability ends up between 0 and 1, they add up to 1, and a higher score always gets a higher probability. You do not need its formula. You need one property of it: the ratio between two probabilities depends only on the gap between their two scores. Each point of gap multiplies the ratio by about 2.7.

Here is a real table. It was measured for this course on the laptop model, Qwen3-0.6B in bf16 on the Apple M1 Max, with a prompt of the same shape as the herb prompt: "Name one fruit. Answer with one word.", through the chat template with thinking off.

In the 1.0 column, "Apple" is most likely, at 0.574. "Mango" and "Banana" are not single tokens in this vocabulary; they compete through their first pieces, M and Ban. Lowercase apple is a different token, with its own score.

Check the gap rule on the top two rows:

score gap, Apple vs M      22.38 - 20.88 = 1.50
measured ratio at 1.0      0.574 / 0.128 = 4.5

A gap of 1.5 points makes Apple 4.5 times as likely as M.

Temperature

Temperature is a number the sampler divides every score by, before softmax. Before you look at the other columns, predict. Raise the temperature from 1.0 to 1.5. Does the top token become more likely or less?

Work it out with the gap rule:

temperature 0.5   scores / 0.5   44.76 and 41.76   gap 3.00
                  ratio about 2.7 x 2.7 x 2.7 = 19.7
                  measured: 0.914 / 0.046 = 19.9
 
temperature 1.0   scores / 1.0   22.38 and 20.88   gap 1.50
                  measured: 0.574 / 0.128 = 4.5
 
temperature 1.5   scores / 1.5   14.92 and 13.92   gap 1.00
                  ratio about 2.7
                  measured: 0.289 / 0.106 = 2.7

Less likely. Dividing by a small number stretches every gap, so the top token takes almost everything: 0.914 at temperature 0.5. Dividing by a big number shrinks every gap, so probability spreads to tokens further down the list: Apple falls to 0.289 at 1.5.

Now the same measurement for the herb prompt, with one surprise first. On Qwen3-0.6B, the top next token for this prompt is tom, the start of "tomato", at 0.547, and five of ten draws at temperature 1.0 answer "tomato". Tomato is not a herb. A model computes scores for what tends to come next; it does not check facts. So the herb runs in this lesson use its bigger sibling, Qwen3-1.7B, in a version that stores each weight in 4 bits instead of 16, through MLX, Apple's machine learning library for its own chips (so this script needs an Apple silicon Mac). It prints the top 6 next tokens for Priya's question at three temperatures:

import mlx.core as mx                      # mlx 0.32.4, Apple silicon only
from mlx_lm import load, generate          # mlx-lm 0.32.2
from mlx_lm.sample_utils import make_sampler
 
model, tok = load("mlx-community/Qwen3-1.7B-4bit")
 
messages = [{"role": "user",
             "content": "Name one herb to add to tomato soup. Answer with one word."}]
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)
ids = tok.encode(prompt, add_special_tokens=False)
 
scores = model(mx.array(ids)[None])[0, -1].astype(mx.float32)
 
print("scores in one pass:", scores.shape[0])
top = mx.argsort(-scores)[:6].tolist()
for t in [0.5, 1.0, 1.5]:
    probs = mx.softmax(scores / t, axis=-1)
    print(f"T={t}: " + "  ".join(f"{tok.decode([i])!r} {probs[i].item():.3f}" for i in top))

load downloads the model and its tokenizer. apply_chat_template wraps the question in Qwen3's chat template, as in lesson 1; enable_thinking=False asks for a direct answer with no thinking tokens (output tokens the model writes before the answer). tok.encode turns the string into token IDs. model(...) runs one forward pass over the whole prompt; [0, -1] takes the scores at the last position, the only one that decides the next token. mx.argsort(-scores)[:6] finds the six highest. For each temperature, scores / t divides every score and mx.softmax turns the result into probabilities.

Run on an Apple M1 Max laptop:

scores in one pass: 151936
T=0.5: 'C' 0.227  'Ch' 0.227  'Bay' 0.177  'O' 0.137  'Th' 0.137  'Rose' 0.031
T=1.0: 'C' 0.139  'Ch' 0.139  'Bay' 0.122  'O' 0.108  'Th' 0.108  'Rose' 0.051
T=1.5: 'C' 0.076  'Ch' 0.076  'Bay' 0.070  'O' 0.064  'Th' 0.064  'Rose' 0.039

Read it like the fruit table: the top row, the gap to the second, and how each temperature moves probability between them. No herb dominates the way Apple did. The top token has 0.139 at temperature 1.0, against Apple's 0.574, and C and Ch have the same probability to three decimal places. The tokens are first pieces again: Th can become "Thyme", O "Oregano", Rose "Rosemary", and only the next passes decide. At 0.5 the top five tokens hold 0.905 of the probability; at 1.5 all six together hold only 0.389.

You might expect a lower temperature to make the model smarter, and a higher one more creative. It is wrong. Between the columns of the fruit table, only the sampler's division changed. The scores, the weights and the pass were identical. OpenAI's API reference describes the knob correctly: "Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic." Random or focused, not smarter or dumber. A low temperature can even hurt. Qwen3's model card warns, for its thinking mode: "DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions." The right model: temperature reshapes the probabilities the model already produced; it never changes the model.

The Draw

Now the sampler draws one token at random, weighted by the probabilities. At temperature 1.0, in 100 draws Apple comes out about 57 times, M about 13, apple about 10. If M is drawn, the next pass continues it, most likely into "Mango". These lines continue the same script and ask the herb question ten times at temperature 1.0:

sampler = make_sampler(temp=1.0)           # plain temperature 1.0: no top-p, no top-k
for seed in range(10):
    mx.random.seed(seed)
    print(generate(model, tok, prompt, max_tokens=5, sampler=sampler))

make_sampler(temp=1.0) builds a sampler that draws from the plain temperature 1.0 probabilities, with no top-p or top-k cut (below). mx.random.seed(seed) fixes the random draw, so each answer can be repeated. generate runs the loop from lesson 1, a pass, a draw, append and again, for at most 5 tokens, enough for a one-word herb.

Cumin
Cumin.
Thyme.
Thyme.
Bay leaves
Garlic.
Dill.
Clove
Thyme.
Thyme.

Same scores on every run. Different draws. "Thyme" comes out four times. C became "Cumin" twice and "Clove" once, never "Chives". "Garlic" and "Dill" start with pieces outside the top six (Gar and D): at temperature 1.0 the top six hold 0.667 of the probability, so about one draw in three lands further down. And cumin, garlic and clove are spices, not herbs. Again, the model scores what tends to follow; it does not check what is true.

Cutting the Tail: Top-p and Top-k

Below the six rows shown sit about 151,930 more tokens, each with a tiny probability. Together they are not tiny. At temperature 1.5 the top six fruit tokens add up to only about 0.62, so more than a third of the draws land further down the list, where the text can stop making sense.

Holtzman and others named this in 2019. They showed that always taking the likeliest token "leads to text that is bland and strangely repetitive", and proposed instead "sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution". That method is top-p, also called nucleus sampling: keep the smallest set of top tokens whose probabilities add up to at least p, drop the rest, and scale the kept ones so they add up to 1 again.

Top-p 0.8 on the fruit table at temperature 1.0:

add the top tokens until the total reaches 0.8
  Apple                          0.574                total 0.574
  M                              0.128                total 0.702
  apple                          0.100                total 0.802   reached 0.8, stop
 
keep 3 tokens out of 151,936; drop everything else
scale the kept ones to add up to 1 (divide by 0.802)
  Apple   0.574 / 0.802 = 0.716
  M       0.128 / 0.802 = 0.160
  apple   0.100 / 0.802 = 0.125

Now the draw can give only one of three tokens. Top-p adapts: when one token dominates it keeps very few, and when probabilities are spread out it keeps many.

Top-k keeps a fixed number, the k most likely tokens. Min-p keeps every token with at least a set fraction of the top token's probability: min-p 0.1 here keeps tokens at or above 0.0574, the same three.

Most engines run these steps in this order (details differ by engine):

scores from the pass
  -> adjustments (biases, penalties, a grammar mask)
  -> divide by temperature
  -> softmax: probabilities
  -> cut the tail (top-k, top-p, min-p), scale back to 1
  -> draw one token

A model also ships with default settings. Qwen3-0.6B has a second small file, generation_config.json. These are its sampling fields:

{
  "do_sample": true,
  "temperature": 0.6,
  "top_k": 20,
  "top_p": 0.95
}

An app that never sets a temperature gets this file's settings: sample at random, temperature 0.6, keep at most the top 20 tokens, then the top-p 0.95 nucleus.

Temperature Zero

Here is the question to stop and think about. Set the temperature to zero. Will the answer be the same every time?

...

Dividing by zero is not defined, so engines treat temperature 0 as greedy: skip the draw and take the highest score. On Priya's laptop, with one prompt running alone, the answer is yes. The research for this course ran Qwen3-0.6B twice on the same prompt on the M1 Max and measured a largest score difference of exactly 0.0. Same scores, same top token, same answer.

On a provider's server, the answer is "not always". Anthropic's API reference says so directly: "Note that even with temperature of 0.0, the results will not be fully deterministic." The reason is not in the sampler but in how a server shares a GPU between many people's calls; lesson 6 shows how.

Logprobs

Some APIs return the probability of each token they wrote as a logprob: the log of the probability. For Apple at 0.574 the logprob is -0.555; near 0 means likely, very negative means unlikely. OpenAI's API returns them when you set logprobs (and up to 20 alternatives per position with top_logprobs). Anthropic's Messages API has no logprobs parameter (checked 2026-10-18). Engineers use them as confidence scores, for example in classification.

Check today. On 2026-10-18, Anthropic's API reference said models released after Claude Opus 4.6 reject any temperature other than 1.0 and any top_k, and accept only top_p values of 0.99 or more. OpenAI's reference marked seed as "Deprecated." Closed APIs are removing sampling knobs as new model generations ship, so what you may set depends on the model. Check the parameter list: https://platform.claude.com/docs/en/api/messages/create

The model ends at scores; the sampler turns them into probabilities, reshapes them with temperature, cuts the tail with top-p or top-k, and draws, so the choice between two answers happens after the model, not inside it.

What One Token Costs to Compute

How much arithmetic is one pass? Count for Llama 3.1 8B. In a matrix multiply, each weight is used once per token, for one multiply and one add. So the rule is about 2 arithmetic operations per parameter per token. Kaplan and others give the same rule in their 2020 scaling-laws paper, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication".

The embedding table is the exception: a lookup costs no arithmetic.

all parameters                         8,030,261,248
minus the embedding table               - 525,336,576   (a lookup)
parameters used in arithmetic          7,504,924,672
 
x 2 operations each                   15,009,849,344
                                      about 15.0 billion operations per token

Fifteen billion operations for every token, in every pass. You will also see these counted as FLOPs (floating-point operations).

Attention Adds Work That Grows

That 15.0 billion is the weights' share. Attention adds more, and its share depends on how many tokens came before. For each earlier token, the new token compares its query with a key and mixes in a value:

compare with one earlier key      4,096 multiplies + 4,096 adds   =   8,192
mix in that token's value         4,096 multiplies + 4,096 adds   =   8,192
per earlier token, per layer                                      =  16,384
x 32 layers                                                       = 524,288

So each earlier token adds 524,288 operations. For the last token of Cookvale's 12,060-token turn 1:

attention    524,288 x 12,060              = 6,322,913,280   about 6.3 billion
weights                                     = 15,009,849,344  about 15.0 billion
attention as a share of the weights' work    6.3 / 15.0       = 42 percent

When does attention catch up with the weights?

15,009,849,344 / 524,288 = 28,629   about 28,600 tokens

Below about 28,600 tokens of context, the weights do most of the arithmetic on this model. Above it, attention does. Cookvale's chats stay well below that point. At 131,072 tokens, the longest Llama 3.1 8B accepts, attention is more than four times the weights' share for that token. This is why very long prompts get expensive faster than their token count suggests.

One Answer Is Many Passes

Now the loop. A Cookvale answer averages 300 tokens. Each token needs its own pass, and each pass runs all 32 layers. Leviathan and others state it in their 2023 paper on speculative decoding: "decoding K tokens takes K serial runs of the model." Serial means one after another: pass 2 cannot start until pass 1 has chosen its token, because pass 2 reads that token.

passes for a 300-token answer       300
layer runs                          300 x 32 = 9,600
 
arithmetic, weights                 300 x 15.0 billion             = 4.50 trillion
arithmetic, attention               300 x 524,288 x about 12,210
                                    (average tokens before each)   = 1.92 trillion
total for the answer                                               = about 6.4 trillion operations

Is 6.4 trillion a lot? An NVIDIA H100 lists 989 trillion operations per second for these numbers:

6.4 trillion / 989 trillion per second = 0.0065 s, about 6.5 thousandths of a second

Yet lesson 1 put these 300 tokens at about 1.6 seconds on that GPU. Something other than arithmetic sets the speed of the 300 passes. Lesson 4 finds it.

Each new token costs about two operations per parameter plus a little more for every earlier token, and an answer of 300 tokens is 300 of those passes, one after another.

Forcing the Shape of the Output

The sampler has one more power. It can refuse tokens.

Cookvale's assistant has a tool called add_to_shopping_list. When a user says "put basil on my list", the model does not call anything itself. It writes the tool's arguments as text, token by token, and the app reads that text as JSON. Here is the tool, in Anthropic's tool format, with strict tool use turned on:

{
  "name": "add_to_shopping_list",
  "description": "Add ingredients to the user's shopping list.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "items": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "quantity": { "type": "number" },
            "unit": {
              "type": "string",
              "enum": ["g", "kg", "ml", "l", "tsp", "tbsp", "piece", "bunch"]
            }
          },
          "required": ["name", "quantity", "unit"],
          "additionalProperties": false
        }
      }
    },
    "required": ["items"],
    "additionalProperties": false
  }
}

input_schema is a JSON Schema: items is a list, each item has a name (text), a quantity (a number) and a unit that must be one of eight values, and nothing else is allowed.

Without help, nothing stops the model from writing something else: every vocabulary token has a score at every step. A comma can end a list, or the quantity can come out as the word "a". OpenAI's launch post for this feature says it plainly: "they are generally free to sample a curly brace token at any time, even when that would not produce valid JSON."

The Mask

The fix is constrained decoding. Before the sampler draws, a grammar built from the schema checks every token in the vocabulary. Every token that would break the schema gets its score set to minus infinity. After softmax, a score of minus infinity has a probability of exactly 0: its gap to every other token is infinite. So only legal tokens can be drawn. The model and its pass are unchanged. Only the menu shrinks.

That is the "grammar mask" in the sampler order from the previous section. Follow it through the call for "put basil on my list":

written so far                                  the grammar allows next
(nothing)                                       an opening brace (or a space)
{"items": [{"name": "                           any text, or a quote to end it
{"items": [{"name": "basil", "quantity":        a space or the start of a number
{"items": [{"name": "basil", "quantity": 1,
  "unit": "                                     only tokens that begin one of:
                                                g kg ml l tsp tbsp piece bunch
{"items": [{"name": "basil", "quantity": 1,
  "unit": "bunch"}]                             a closing brace, then the end

Suppose that after "quantity": the highest-scoring token is " followed by a, as if the model wanted to write "a bunch". Both are illegal there. The mask sets their scores to minus infinity, and the sampler draws from the tokens that start a number. The model writes 1.

OpenAI describes its own implementation in the same terms: "after every token, our inference engine will determine which tokens are valid to be produced next based on the previously generated tokens and the rules within the grammar that indicate which tokens are valid next. We then use this list of tokens to mask the next sampling step, which effectively lowers the probability of invalid tokens to 0." Anthropic's documentation says strict tool use exists "To enforce JSON Schema compliance on tool inputs with grammar-constrained sampling".

Why bother, if a good model writes valid JSON most of the time? OpenAI's launch post gives the numbers on its own schema benchmark (checked 2026-10-18):

Training got the model to 93 percent. In the post's words, that "still did not meet the reliability that developers need", so they "took a deterministic, engineering-based approach to constrain the model's outputs to achieve 100% reliability."

Older JSON mode is weaker. OpenAI's docs: "JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors." It allows any valid JSON. A schema mask allows only your shape.

Tokens Do Not Line Up With Grammar

A grammar works in characters; the model writes tokens. In the o200k_base tokenizer, {" is a single token, an opening brace and a quote together, and so are ": and "}. Each crosses a grammar boundary. So at every step the engine must check the full text of each candidate token against the grammar's current state, for all 150,000 to 200,000 tokens. Open-source engines such as XGrammar and llguidance make this fast with precomputed masks; llguidance reports "on the order of 50μs of CPU time per token (for 128k tokenizer)".

What It Costs, and What It Cannot Do

The mask is not free:

• A first-call delay. The schema is compiled into a grammar first. OpenAI's launch post: "Typical schemas take under 10 seconds to process on the first request, but more complex schemas may take up to a minute." Anthropic: "Compiled grammars are cached for 24 hours from last use".
• Escape cases. Anthropic's docs list two ways a response can still break the schema: a refusal (stop_reason: "refusal") and running out of tokens (stop_reason: "max_tokens"). A list cut off at the token limit is not valid JSON, mask or not.
• Schema features. Not every JSON Schema rule can become a grammar. Anthropic's list of unsupported features includes "Numerical constraints (such as minimum, maximum, multipleOf)".

Now a prediction. Strict tool use is on for add_to_shopping_list. Can the model still put a wrong value in a valid field?

Yes. {"name": "oregano", "quantity": 1, "unit": "bunch"} is valid when the user asked for basil. So is a quantity of 100. The mask checks shape, not truth: among the legal tokens, the sampler still picks by the model's scores. Your code still checks that 100 bunches of basil makes sense.

Whether constraints lower quality is argued. Tam and others reported in 2024 that "stricter format constraints generally lead to greater performance degradation in reasoning tasks"; the makers of the Outlines library replied that the evaluation was flawed. Either way, when an answer needs reasoning, put a free-text reasoning field before the answer field in the schema.

Constrained decoding sets the score of every illegal token to minus infinity before the draw, so the output always has the schema's shape, but the values inside it are still the model's choice.

The Puzzle, Answered

What does the model compute? For each new token, one forward pass: a row lookup, then every layer (attention mixes in the earlier tokens, the MLP works on each token alone), then one score for every token in the vocabulary, 151,936 of them for Qwen3-0.6B. Both of Priya's runs computed exactly the same scores, because on one laptop with one prompt the pass gives the same numbers every time.

The choice happened after the model, in the sampler. Priya's runs drew at temperature 1.0 with nothing cut. Th, the start of "Thyme", has a probability of 0.108. "Dill" starts with D, which is not even in the top six, yet one draw landed on it. Same scores, two different draws. With no temperature set, Qwen3's default settings in generation_config.json still sample at random (temperature 0.6, top-k 20, top-p 0.95), so the answer would still change from run to run, only among fewer candidates. At temperature zero, her laptop would give the same herb every time.

The Takeaway

1. A model file is a table of parameters; config.json gives its shape. Rows times columns gives the count, times 2 the bytes in bf16: Llama 3.1 8B is 8,030,261,248 parameters, 16.06 GB.
2. A pass starts with a lookup: the token ID selects a row of the embedding table.
3. Every layer has attention and an MLP. Attention compares each token with itself and every earlier token and takes a weighted mix; it is the only place tokens exchange information, and it never looks forward. The MLP works on each token alone and holds 81 percent of each layer in Llama 3.1 8B.
4. The pass ends with one score for every vocabulary token, never a word.
5. The sampler turns scores into probabilities. Temperature divides the scores first (low stretches the gaps, high shrinks them) and never changes the model. Top-p and top-k cut the tail. Then one token is drawn. Temperature zero takes the top score.
6. Each token costs about 2 operations per parameter, 15.0 billion for Llama 3.1 8B, plus 524,288 for every earlier token. A 300-token answer is 300 passes, one after another.
7. Constrained decoding sets illegal tokens to minus infinity before the draw: the output always has the schema's shape, but the values are still the model's choice.

The one sentence to keep: for each new token the model runs every layer once and ends with a probability for every token in its vocabulary; the sampler, not the model, makes the final choice.

An open question for the next lesson: each pass uses every weight of the model, all sixteen gigabytes of them for Llama 3.1 8B. The arithmetic of a whole 300-token answer would take a GPU a few thousandths of a second, yet the answer takes more than a second to stream. How fast can the GPU's memory deliver those sixteen gigabytes, again and again, once for every token? Prefill and decode.

References

Papers:

• Vaswani et al., "Attention Is All You Need", NeurIPS 2017: the transformer, scaled dot-product attention (Equation 1), the mask that stops positions "from attending to subsequent positions"
• Kaplan et al., "Scaling Laws for Neural Language Models", 2020: about 2 operations per parameter per token, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication"
• Holtzman et al., "The Curious Case of Neural Text Degeneration", ICLR 2020: nucleus (top-p) sampling
• Nguyen et al., "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs", 2024: min-p
• Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023: "decoding K tokens takes K serial runs of the model"
• Tam et al., "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models", 2024; reply by Will Kurt, ".txt: Say What You Mean"
• Dong et al., "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", 2024

Model files on Hugging Face (checked 2026-10-18):

• Qwen/Qwen3-0.6B, revision c1899de289a04d12100db370d81485cdf75e47ca: config.json, generation_config.json, model.safetensors (1,503,300,328 bytes), parameter count 751,632,384 as stored, 596,049,920 unique
• Llama 3.1 8B Instruct config.json, read from the unsloth/Llama-3.1-8B-Instruct mirror (Meta's repository is gated); parameter count 8,030,261,248
• Qwen3 model card: "DO NOT use greedy decoding" in thinking mode

API documentation (checked 2026-10-18):

• Anthropic Messages API reference: temperature not fully deterministic at 0.0; temperature, top_p and top_k restrictions for models released after Claude Opus 4.6; no logprobs parameter
• Anthropic structured outputs: constrained decoding, strict tool use, grammar caching for 24 hours, refusal and max_tokens cases, unsupported numerical constraints
• OpenAI Chat Completions reference: temperature, top_p, logprobs, top_logprobs, seed marked "Deprecated."
• OpenAI Structured Outputs guide: JSON mode vs structured outputs
• OpenAI, "Introducing Structured Outputs in the API", 2024-08-06: the token mask, the context-free grammar, 93 vs 100 percent, first-request latency
• llguidance README: about 50μs of CPU time per token

Measurements:

• The herb table and the ten herb samples: mlx-community/Qwen3-1.7B-4bit (Qwen3-1.7B with 4-bit weights) on Apple M1 Max, mlx 0.32.4, mlx-lm 0.32.2, measured for this course. The "tomato" result: Qwen3-0.6B on the same laptop's CPU in float32, same prompt, same three temperatures and ten seeded draws
• The parameter count and file size output: Qwen3-0.6B, transformers 5.20.1, torch 2.14.1, measured for this course
• The fruit next-token table and the run-to-run 0.0 score difference: Qwen3-0.6B in bf16 on Apple M1 Max (MPS), transformers 5.20.1, torch 2.14.1, measured for this course on 2026-10-18
• The {", ": and "} tokens: tiktoken 0.14.0, o200k_base, measured for this course on 2026-10-18
• The scripts in this lesson: Python 3.12; the parameter count with transformers 5.20.1 and torch 2.14.1 on Qwen/Qwen3-0.6B; the herb script with mlx 0.32.4 and mlx-lm 0.32.2 on mlx-community/Qwen3-1.7B-4bit