Inside the forward pass

Priya wants to see what a model actually is, so she downloads an open model, Qwen3-0.6B, from Hugging Face. The biggest file is model.safetensors, about 1.5 GB. Next to it sits config.json, 726 bytes. The rest are tokenizer files.

She also downloads its bigger sibling, Qwen3-1.7B, in a compressed version of about 1 GB. She runs it on her laptop and asks the same question twice: "Name one herb to add to tomato soup." The first time it answers "Thyme". The second time it answers "Dill".

Same file, same prompt, same laptop. Nothing in the file changed between the two runs.

So here is the puzzle for this lesson. What does the model compute when it reads a prompt? And where, exactly, does the choice between "Thyme" and "Dill" happen?

This lesson answers the first question; the next one answers the second. By the end of it you will be able to work out a model's size from its config.json and follow one token through a pass.

Two facts from earlier lessons carry this one. What happens in one model call showed that the model reads one long string of tokens and produces one new token per pass. Tokens: what you pay for showed that a token is a piece of text stored as an integer, its token ID, and that the vocabulary is the full list of tokens a tokenizer knows. This lesson opens the pass itself.

What Is in a Model File

A model is a file of numbers. Each number is a parameter. Together, all the parameters are the model's weights. Training is the process that set them, long before Priya downloaded the file. While the model runs, they never change.

The file does not say how the numbers are arranged. That is the job of config.json. Here are its main fields for Qwen3-0.6B, copied from the file on Hugging Face:

{
  "head_dim": 128,
  "hidden_size": 1024,
  "intermediate_size": 3072,
  "num_attention_heads": 16,
  "num_hidden_layers": 28,
  "num_key_value_heads": 8,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "vocab_size": 151936
}

Each field is a size:

• vocab_size: 151,936. The number of tokens the model can read and write.
• hidden_size: 1,024. Inside the model, every token is carried as a list of 1,024 numbers. A list of numbers like this is called a vector.
• num_hidden_layers: 28. A layer is one of the identical blocks a token passes through, one after another.
• num_attention_heads, num_key_value_heads, head_dim: how the attention step inside each layer is split up (next section).
• intermediate_size: 3,072. How wide the MLP step inside each layer is (next section).
• torch_dtype: bfloat16, usually written bf16: a 16-bit number format. Each parameter takes 2 bytes.
• tie_word_embeddings: true. One table is used at both the start and the end (this matters for the file size, below).

Llama 3.1 8B, the model this course uses for GPU arithmetic, has the same fields with bigger values in its config.json: hidden size 4,096, 32 layers, 32 attention heads, 8 key and value heads of 128, MLP width 14,336, a vocabulary of 128,256, and no shared table.

From these sizes alone you can count every parameter in the model. Before you do, predict one thing. Each layer has two main steps, attention and the MLP. Attention is the part named in the title of the paper that introduced this design. Which step holds more of the parameters?

...

Count them. Every weight table in the model is a grid of numbers, so its size is rows times columns. This is the count for Llama 3.1 8B, from research done for this course and checked against Hugging Face's own total:

embedding table   128,256 x 4,096                    =   525,336,576
 
per layer:
  attention
    query    4,096 x (32 x 128)                      =    16,777,216
    key      4,096 x ( 8 x 128)                      =     4,194,304
    value    4,096 x ( 8 x 128)                      =     4,194,304
    output   (32 x 128) x 4,096                      =    16,777,216
    attention subtotal                               =    41,943,040
  MLP: gate, up and down, 3 x 4,096 x 14,336         =   176,160,768
  two norms, 2 x 4,096                               =         8,192
  layer total                                        =   218,112,000
 
32 layers         32 x 218,112,000                   = 6,979,584,000
final norm                                           =         4,096
output table      4,096 x 128,256                    =   525,336,576
 
TOTAL                                                = 8,030,261,248

The answer to the prediction is in the middle of that block:

MLP share of one layer        176,160,768 / 218,112,000 = 0.81 = 81 percent
attention share of one layer   41,943,040 / 218,112,000 = 0.19 = 19 percent

You might expect attention to be most of the model. It is wrong. In Llama 3.1 8B the MLP holds 81 percent of every layer's parameters, about four times as many as attention. Attention matters for other reasons: it is the only step where tokens exchange information, and its cost grows with the length of the prompt. The right model: most of the weights sit in the MLPs; attention is small in parameters and large in importance.

So the model has three bands. An embedding table at the start, 525 million parameters. Thirty-two identical layers in the middle, 6.98 billion. An output table at the end, 525 million. The layers are 87 percent of the model.

Now the bytes. Each bf16 parameter is 2 bytes:

Llama 3.1 8B   8,030,261,248 parameters x 2 bytes = 16,060,522,496 bytes = 16.06 GB

A GB in this course is 1,000,000,000 bytes. You will meet 16.06 GB again: it is what a GPU must hold before it can answer anyone.

Why Priya's File Is 1.5 GB, Not 1.2

The same count for Qwen3-0.6B, shorter:

embedding table   151,936 x 1,024                    =   155,582,464
one layer
  attention (query 1,024 x 2,048, key and value
    1,024 x 1,024 each, output 2,048 x 1,024,
    plus two small norms of 128)                     =     6,291,712
  MLP, 3 x 1,024 x 3,072                             =     9,437,184
  two norms, 2 x 1,024                               =         2,048
  layer total                                        =    15,730,944
28 layers         28 x 15,730,944                    =   440,466,432
final norm                                           =         1,024
output table      the embedding table, used again    =             0
TOTAL                                                =   596,049,920
 
bytes in bf16     596,049,920 x 2                    = 1,192,099,840 = 1.19 GB

In this small model the MLP is 60 percent of each layer (9,437,184 / 15,730,944), still the largest part. The vocabulary table is 26 percent of the whole model, which is why small models use one table for both ends.

But the file on Hugging Face is 1,503,300,328 bytes, about 1.50 GB. Where do the extra 0.31 GB come from? The file stores the shared table twice, once under each name, so Hugging Face's own counter reports 751,632,384 parameters. The few bytes left over are the file's header, a short JSON list of every table's name, shape and position:

596,049,920 + 155,582,464 (the table, a second time)  =   751,632,384
751,632,384 x 2 bytes                                 = 1,503,264,768
file size                                             = 1,503,300,328
difference (the file's header)                         =        35,560 bytes

When the model loads, the two copies become one table in memory. So "how big is the model" has two honest answers here: 1.19 GB of unique weights, 1.50 GB on disk.

You can count the parameters yourself. This needs Python 3.12 with torch 2.14.1 and transformers 5.20.1; the first run downloads the model, about 1.5 GB:

import os
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype=torch.bfloat16)
 
total = sum(p.numel() for p in model.parameters())
print(f"parameters:    {total:,}")
print(f"bytes in bf16: {total * 2:,}")
 
path = hf_hub_download("Qwen/Qwen3-0.6B", "model.safetensors")
print(f"file on disk:  {os.path.getsize(path):,}")

from_pretrained reads config.json, builds the layers it describes, and fills them from model.safetensors. model.parameters() lists every weight table once, so the shared table counts once. p.numel() is the number of values in one table. hf_hub_download returns the path of the downloaded file, and os.path.getsize reads its size in bytes.

Run on an Apple M1 Max laptop:

parameters:    596,049,920
bytes in bf16: 1,192,099,840
file on disk:  1,503,300,328

The count matches the worked total exactly, and the file is the 1.50 GB from above, with the shared table stored twice.

A model file is a table of numbers whose shape the config describes; count rows times columns and you have the parameters, and times two the bytes.

One Token, One Pass

Now follow one token through Llama 3.1 8B: the last token of the herb prompt, after the chat template from the first lesson has wrapped the question.

Step 1: Look Up a Row

The token arrives as an integer, its token ID. The model uses that integer as a row number in the embedding table. In Llama 3.1 8B the table has 128,256 rows, one per vocabulary token, and 4,096 numbers in each row. The row becomes the token's vector.

This step is a lookup, with no arithmetic in it. That matters when we count the work.

Step 2: Run Every Layer

The vector now goes through 32 layers, one after another. Each layer reads the vector and adds an update to it, so the vector is a running total of the original row plus every layer's update. Each layer has two steps.

Attention. This is where a token looks at the tokens before it. In plain words: each token compares itself with every earlier token and takes a weighted mix of what it finds. The weight tables from the count do this work:

1. From its vector, the token computes three shorter lists of numbers with the query, key and value tables from the count: a query (what it is looking for), a key (what it offers to later tokens) and a value (what it hands over if chosen).
2. The token compares its query with the key of itself and of every earlier token. Each comparison gives one number: how well the two match.
3. Those numbers are turned into weights that add up to 1. A strong match gets a big weight.
4. The token takes the value of every earlier token, multiplies each by its weight, and adds them up. That mix is its update. The output table turns the mix back into 4,096 numbers.

Attention runs in several heads side by side, each with its own query, key and value, so each can compare tokens in a different way. Llama 3.1 8B has 32 query heads of 128 numbers each (32 x 128 = 4,096). It has only 8 key and value heads: each group of 4 query heads shares one. What the KV cache costs explains why that sharing matters so much.

Nobody writes these comparisons by hand; training set the tables that produce queries, keys and values. When the last token of the herb prompt runs attention, it can put large weights on "herb" and "tomato soup" and small ones on the template's marker tokens. That is how the whole prompt reaches the position that chooses the next token.

Two properties of attention matter for the rest of this course.

• It is the only step where tokens exchange information. Everything else in a layer works on each token alone.
• It only looks back. A token compares itself with itself and earlier tokens, never with later ones. Models built this way are called causal. The paper that introduced this architecture says it modifies attention "to prevent positions from attending to subsequent positions." So when a new token arrives, nothing about the earlier tokens changes, because they never looked at it. The KV cache builds on this fact.

Here is attention as the original paper writes it (Vaswani et al., "Attention Is All You Need", 2017, Equation 1):

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

You do not need this formula for anything in this course. The four steps above say the same thing in words.

The MLP. The second step works on each token alone. MLP stands for multi-layer perceptron; in this course it means a block of matrix multiplies. In Llama's MLP, the first table turns 4,096 input numbers into 14,336 output numbers, and each output number is a weighted sum of all 4,096 inputs:

one output number = input 1 x weight 1 + input 2 x weight 2 + ... + input 4,096 x weight 4,096
                  = 4,096 multiplies and 4,096 adds
the 4,096 weights are parameters, used once each
14,336 output numbers x 4,096 weights = 58,720,256 parameters in this table

Llama's MLP has three tables of this size, called gate, up and down. Gate and up each turn 4,096 numbers into 14,336, their results are combined, and down turns the 14,336 back into 4,096: the 176 million parameters per layer from the count. Because most of the weights are here, most of a pass's arithmetic is here too. (Each step also starts with a small norm, a rescaling with 4,096 parameters: the "two norms" in the count.)

Step 3: Score Every Token

After layer 32, one more norm, then the output table. It has one column of 4,096 weights for every token in the vocabulary. Each token's score is the weighted sum of the final vector with that token's column:

score for one vocabulary token = 4,096 multiplies and 4,096 adds
scores produced                = 128,256, one per vocabulary token

Papers call these scores logits; this course says "score". A high score marks a likely next token, a low score an unlikely one. The model does not produce a word. It produces 128,256 numbers (151,936 on the laptop model, one per token of Qwen3's vocabulary).

A Whole Prompt at Once

In the first pass, the one the pause, the rush and the bill called prefill, all the prompt's tokens go through the layers together, and in attention each looks back at the tokens before it. Only the scores at the last position decide the next token. The chosen token is added to the end, and the next pass runs for that one new token. That is the forward pass: one run through every layer, from lookup to scores.

Apply it to Cookvale. Turn 1 is 12,060 tokens: 2,000 of tools, 6,000 of system prompt, 4,000 of recipe and a 60-token question. In prefill, the last token runs attention against all 12,059 before it, so the recipe, the house rules and the question all reach the position that chooses the first word of the answer. A token in the middle of the recipe sees the tools and the system prompt, but never the question, because the question comes after it.

One pass turns a token into a vector by lookup, sends it through every layer (attention mixes in earlier tokens, the MLP works on each token alone), and ends with one score for every token in the vocabulary.

The Puzzle, Answered

What does the model compute? For each new token, one forward pass: a row lookup, then every layer (attention mixes in the earlier tokens, the MLP works on each token alone), then one score for every token in the vocabulary, 151,936 of them for Qwen3-0.6B. Both of Priya's runs computed exactly the same scores, because on one laptop with one prompt the pass gives the same numbers every time.

Where the choice between "Thyme" and "Dill" happens is the next lesson's question.

The Takeaway

1. A model file is a table of parameters; config.json gives its shape. Rows times columns gives the count, times 2 the bytes in bf16: Llama 3.1 8B is 8,030,261,248 parameters, 16.06 GB.
2. A pass starts with a lookup: the token ID selects a row of the embedding table.
3. Every layer has attention and an MLP. Attention compares each token with itself and every earlier token and takes a weighted mix; it is the only place tokens exchange information, and it never looks forward. The MLP works on each token alone and holds 81 percent of each layer in Llama 3.1 8B.
4. The pass ends with one score for every vocabulary token, never a word.

The one sentence to keep: a model file is a table of numbers, and one pass through it ends with a score for every token, not with a word.

An open question to carry forward: same file, same prompt, same laptop, and Priya got "Thyme" once and "Dill" once. If the pass only gives scores, where is the herb chosen? That is the next step, where the next token is chosen.

References

Papers:

• Vaswani et al., "Attention Is All You Need", NeurIPS 2017: the transformer, scaled dot-product attention (Equation 1), the mask that stops positions "from attending to subsequent positions"

Model files on Hugging Face (checked 2026-10-18):

• Qwen/Qwen3-0.6B, revision c1899de289a04d12100db370d81485cdf75e47ca: config.json, generation_config.json, model.safetensors (1,503,300,328 bytes), parameter count 751,632,384 as stored, 596,049,920 unique
• Llama 3.1 8B Instruct config.json, read from the unsloth/Llama-3.1-8B-Instruct mirror (Meta's repository is gated); parameter count 8,030,261,248

Measurements:

• The parameter count and file size output: Qwen3-0.6B, transformers 5.20.1, torch 2.14.1, measured for this course
• The scripts in this lesson: Python 3.12; the parameter count with transformers 5.20.1 and torch 2.14.1 on Qwen/Qwen3-0.6B