What happens in one model call

Priya is the engineer who owns the cooking assistant in Cookvale, a recipe app. The assistant answers questions while a person cooks one recipe: what to substitute, how long to wait, whether something is done. The answer streams into a chat panel, a few words at a time.

The assistant goes live. The first real user stops cooking and types eight words:

Can I use butter instead of oil?

The answer box stays empty for almost a second. Then the words arrive quickly, one small piece after another. Priya's usage log for that call says:

{ "input_tokens": 12060, "output_tokens": 300 }

On the bill line for the call, the input part is eight times the output part. But on the price list, every output token costs five times as much as an input token.

That gives three questions, and this lesson answers all three:

1. The user typed eight words. Where did twelve thousand tokens of input come from?
2. Why was there a pause, and then a rush of words?
3. Why is output priced higher per token, and why is input still the bigger part of the bill?

A token is one piece of text that the model reads or writes, often a short word or part of a word. The model stores each token as a number, its token ID. Lesson 2 shows how text is cut into tokens. For now, a token is the unit the provider counts and charges for.

Running a trained model to get output is called inference, the word in this course's title. In the lessons we say "a model call", because that is what you write in your code.

What the Provider Actually Receives

Priya's code does not send eight words. It sends a JSON request with four kinds of content: tool definitions, a system prompt, the recipe the user is cooking, and the conversation so far. On turn 1, the conversation is only the new question.

Here is the request, trimmed. Long parts are replaced by a description in square brackets. The shape follows the Anthropic Messages API; other providers use other field names for the same parts.

{
  "model": "mid-size-model",
  "max_tokens": 1024,
  "stream": true,
  "tools": [
    {
      "name": "convert_units",
      "description": "Convert an amount between units, for example cups to grams.",
      "input_schema": {
        "type": "object",
        "properties": {
          "amount": { "type": "number" },
          "from_unit": { "type": "string" },
          "to_unit": { "type": "string" }
        },
        "required": ["amount", "from_unit", "to_unit"]
      }
    }
  ],
  "system": "You are the cooking assistant in Cookvale. [voice and house rules, food safety rules, unit conventions, six example exchanges]",
  "messages": [
    {
      "role": "user",
      "content": "[the recipe being cooked, plus three technique notes]\n\nCan I use butter instead of oil?"
    }
  ]
}

model picks the model. max_tokens is the most output tokens the call may produce. stream: true asks for the answer piece by piece. tools lists functions the model may ask Priya's code to run; the real list has eight (find_recipes, get_recipe, scale_recipe, convert_units, substitute_ingredient, set_timer, add_to_shopping_list, get_pantry), and one is shown. system is the system prompt: instructions that are the same for every user. messages is the conversation.

The recipe sits at the start of the first user message. A retrieved recipe means text the app looks up and pastes into the prompt: Cookvale finds the recipe being cooked, adds three short technique notes, and puts them in front of the question.

The Model Reads One String

A model has no idea what a "system prompt" or a "tool" is. It was trained on one long stream of tokens. So before it runs, the provider turns the JSON into one flat string. The program that does this is the chat template: a text format that wraps each message in special marker tokens.

Closed providers do this on their servers and do not publish their templates. Open models publish theirs. Qwen3 is an open model family whose template is a small program in the file tokenizer_config.json on Hugging Face. Here is the shape of the string it builds for Cookvale's request, using the template's own text, with the long parts cut:

<|im_start|>system
You are the cooking assistant in Cookvale. [... rules and examples ...]
 
# Tools
 
You may call one or more functions to assist with the user query.
 
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "convert_units", ...}}
[... seven more tools ...]
</tools>
 
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
[... the recipe and three technique notes ...]
 
Can I use butter instead of oil?<|im_end|>
<|im_start|>assistant

Four things to see in it.

1. <|im_start|> and <|im_end|> are marker tokens. Each is one token with its own ID, marking where a message starts and ends. The word after <|im_start|> says whose message it is. That is all a "role" is: a word inside the string.
2. The tools are text. The template writes every tool definition into the system message as JSON, with a paragraph telling the model how to ask for a call. The model reads those tokens on every call, and the provider bills them as input on every call. Qwen puts the tool text after the system text; other providers put tools first. The order varies. That tools become text does not.
3. The string ends with <|im_start|>assistant and a new line: an open message with no text yet. The model's job is to continue the string from there.
4. The model also has a way to stop. When it writes <|im_end|> (token ID 151645 in Qwen3), the server stops and returns the text. The other way a call stops is reaching max_tokens.

How much does a template add? The script below renders Cookvale's own two messages, a one-line system prompt and the user's question: 24 tokens of message text become 41 tokens after the Qwen3 template. Templates can add more than markers. The Llama 3.1 8B template on Hugging Face writes "Cutting Knowledge Date: December 2023" and "Today Date: 26 Jul 2024" into the system message unless the caller passes a date. On a two-message example of about the same size, that header was 20 of 51 tokens.

You can render a template yourself. This needs no GPU and no API key, and downloads only Qwen3-0.6B's tokenizer files:

pip install "transformers==5.20.1" "tokenizers==0.23.3" "jinja2==3.1.6"

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
 
messages = [
    {"role": "system", "content": "You are Cookvale's cooking assistant. Answer in one or two short sentences."},
    {"role": "user", "content": "Can I use butter instead of oil?"},
]
 
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
print(text)
 
ids = tok(text)["input_ids"]
print(len(ids), "tokens with the template")
print(sum(len(tok(m["content"])["input_ids"]) for m in messages), "tokens of text alone")

from_pretrained downloads the tokenizer and its chat template from the model's Hugging Face page. messages is the same list shape you send to a provider. apply_chat_template runs the template; tokenize=False returns the string so you can read it; add_generation_prompt=True adds the open <|im_start|>assistant line; enable_thinking=False is a Qwen3 option that adds an empty thinking block so the model answers directly. The next line cuts the rendered string into token IDs and counts them. The last line counts each message's text without any template and adds the counts.

Run on an Apple M1 Max laptop:

<|im_start|>system
You are Cookvale's cooking assistant. Answer in one or two short sentences.<|im_end|>
<|im_start|>user
Can I use butter instead of oil?<|im_end|>
<|im_start|>assistant
<think>
 
</think>
 
 
41 tokens with the template
24 tokens of text alone

The template added 17 tokens to two short messages. Thirteen of them are the markers, the role words and the line breaks around them. The other four are the empty thinking block, <think>, a blank line, </think> and another blank line, which enable_thinking=False writes so the model skips straight to the answer.

Every Turn Sends Everything Again

The user keeps cooking and asks a second question. What does the provider receive now?

You might expect only the new question, because the provider "remembers" the chat. It does not. Nothing kept between calls changes what the model reads. Priya's code builds the whole request again: the same tools, system prompt and recipe, the first question, the first answer, and the new question. The template renders all of it, and the model reads all of it. Anthropic's documentation describes the input of a turn the same way:

"Input phase: Contains all previous conversation history plus the current user message"

So "conversation history" is a longer prompt, nothing more. In this course, the prompt is everything the model reads for one call: tools, system prompt, recipe, history and question.

Each later Cookvale turn adds the previous answer (300 tokens) and the new question (60 tokens):

each new turn adds   300 (previous answer) + 60 (new question) = 360 tokens
 
turn 1 input   2,000 + 6,000 + 4,000 + 60    = 12,060 tokens
turn 2 input   12,060 + 360                  = 12,420 tokens
turn 3 input   12,420 + 360                  = 12,780 tokens
turn 4 input   12,780 + 360                  = 13,140 tokens

On turn 4, the model reads 12,780 tokens it already read on turn 3, all over again. Lesson 7 shows how a provider can skip some of that work, and what it charges when it does.

Where the Twelve Thousand Came From

Now the first prediction. The user typed eight words. How many tokens will the provider bill as input?

Take a moment before you read on.

All of them, the whole prompt:

tool definitions (8 tools)                    2,000 tokens
system prompt                                  6,000 tokens
retrieved recipe and technique notes           4,000 tokens
user question                                     60 tokens
                                              ------
turn 1 input                                  12,060 tokens

The user's words are the smallest part. Two thirds of the prompt, the tools and the system prompt (8,000 tokens), is the same for every user. The question band is a planning number: 60 tokens is the average question across the five languages Cookvale's users write in, and these eight English words are fewer. Lesson 2 counts real questions and the marker tokens, which Cookvale's planning numbers count inside the bands above.

Anthropic's API reference warns that its counts will not match the text you can see, because of this rendering step:

"Under the hood, the API transforms requests into a format suitable for the model. The model's output then goes through a parsing stage before becoming an API response. As a result, the token counts in usage will not match one-to-one with the exact visible content of an API request or response."

Recap. The provider receives JSON, but the model reads one long string made by a chat template. That string holds every tool, the system prompt, the recipe and the whole conversation, and it is sent again in full on every turn.

What the Model Does With That String

First the tokenizer, a program that cuts text into tokens, turns the string into token IDs. For Cookvale's turn 1, that is a list of 12,060 integers, the only form the model ever sees.

Then the model runs. A model is a large file of numbers. Each number is a parameter, and together they are the model's weights. Running the model means a fixed series of multiplications and additions between the tokens' numbers and those weights. One run through the whole model is a pass. Lesson 3 opens the model up. For now, treat the pass as a box.

What comes out of one pass is not an answer. It is a list of scores, one for every token the model could write next. The full list of tokens a tokenizer knows is its vocabulary; Qwen3-0.6B's has 151,936 entries, so each pass ends with 151,936 scores. The scores are turned into probabilities that add up to one. Then a small piece of ordinary code, the sampler, picks one token from those probabilities, usually a likely one but not always the most likely. Lesson 3 shows how it picks.

Prediction: the model has the whole prompt. How many tokens does one pass produce?

One. A pass over 12,060 tokens ends with one chosen token, perhaps "Yes".

The Loop

What happens to that one token? It is added to the end of the string, and the model runs again on the string with one more token. That pass ends with scores for the next token, the sampler picks one, it is added, and the model runs again.

pass 1    reads  [12,060 prompt tokens]                    -> writes "Yes"
pass 2    reads  [12,060 prompt tokens] "Yes"              -> writes ","
pass 3    reads  [12,060 prompt tokens] "Yes" ","          -> writes " you"
pass 4    reads  [12,060 prompt tokens] "Yes" "," " you"   -> writes " can"
...
last      the model writes <|im_end|>, or the call reaches max_tokens

The words are only an illustration. The shape is the point: every pass ends with exactly one new token, and that token becomes part of what the next pass reads.

Prediction: the answer is 300 tokens. How many passes?

Three hundred, one per output token.

The Model Does Not Write the Whole Answer at Once

You might expect the model to produce the full answer and then send it in pieces. That is wrong, and the loop shows why. Token 2 cannot be computed before token 1 exists, because token 1 is part of what pass 2 reads. Token 300 depends on all 299 before it. The paper that introduced speculative decoding (Leviathan, Kalman and Matias, ICML 2023) puts the rule in one line:

"decoding K tokens takes K serial runs of the model."

"Serial" means one after another. Lesson 8 shows ways to get more than one token from a pass, but they guess and check, and the loop is still underneath. When words appear one at a time in a chat app, that is not an animation. You are watching the loop run.

Recap. One pass reads the whole string and ends with scores for the next token; the sampler picks one, it is added, and the model runs again. One pass, one new token.

The Pause, Then the Rush

Back to what Priya saw: almost a second of nothing, then a quick stream. The loop explains both.

Pass 1 is not like the others. When it starts, the model has never seen this prompt, so it must process all 12,060 tokens before it can score anything. Because all of them are known in advance, the GPU handles them together, in one large run. That run takes time, and the user sees nothing until it ends with the first output token.

Every later pass has only one new token. Each is small and quick, and as each one ends, one more token can go to the user.

These two parts have names:

• Prefill is the first pass, over the whole prompt at once. It ends when the first output token exists.
• Decode is every pass after it, each adding one new token.

The user feels them as two numbers on a clock:

• Time to first token (often written TTFT) runs from sending the request until the first token arrives. For a long prompt, prefill is most of it. On a real provider it also includes the network and any time the request waits in a queue for a GPU; providers do not publish how that time splits.
• Time per output token (TPOT) is the time between one streamed token and the next: the length of one decode pass. "Stream speed", in tokens per second, is a fine plain name for it.

The pause is prefill. The rush is decode.

How big are the two parts? Providers do not publish their model sizes or hardware, so we use an open model of known size on a known GPU: Llama 3.1 8B (8,030,261,248 parameters) on one NVIDIA H100, a common data center GPU. Lesson 3 derives the amount of arithmetic and lesson 4 the time of one decode pass. Here we use their results:

Llama 3.1 8B on one H100 (open model, arithmetic only)
 
prefill: one pass over 12,060 tokens
  arithmetic needed       about 219 trillion operations
  H100 peak speed         989 trillion operations per second
  time at full speed      219 / 989 = 0.22 s
  time at a realistic     about half of full speed
    half speed            0.22 s x 2 = about 0.44 s
 
decode: 300 passes of one token each
  one pass                about 5.3 ms
  300 passes              300 x 5.3 ms = 1,590 ms = about 1.6 s
 
decode time / prefill time    1.6 s / 0.44 s = about 4

The prompt is about 40 times longer than the answer (12,060 / 300 = 40.2). Yet the 300 output tokens take about four times as long as the 12,060 input tokens. Priya's "almost a second" comes from a larger model on hardware she cannot see, plus the network and the queue, but the shape is the same: a wait for the whole prompt, then a steady stream. Databricks's engineers state the practical rule in their guide to serving models:

"Output length dominates overall response latency"

The course measured both parts on a laptop, with Qwen3-0.6B, a small open model, and three prompts shaped like Cookvale's: a short system prompt with a recipe summary, a short system prompt with the full recipe, and the full turn-1 prompt with tools, system prompt, recipe and notes. Each ends in the butter question and gets 300 new tokens, always taking the highest-scoring token. The run uses MLX, Apple's machine learning library for its own chips. A model this small on a laptop is limited partly by the fixed cost of starting each pass, so read the numbers as a picture of the two phases, not of a data center. Lesson 4 shows the script and explains what limits each phase.

Run on an Apple M1 Max laptop:

engine=mlx-lm model=Qwen/Qwen3-0.6B greedy, 300 new tokens (end-of-turn tokens banned so every run writes 300)
prompt | prompt tok |  TTFT ms | ms per output token | output tok/s | tokens out | peak GB
 small |        117 |       44 |                 7.5 |        134.7 |        300 |    1.41
medium |      1,010 |      197 |                 7.6 |        132.4 |        300 |    1.91
 large |     12,327 |     7540 |                12.6 |         79.5 |        300 |    3.19

The large prompt is 12,327 tokens here because Qwen3's tokenizer cuts the same text differently from the provider's, which bills Cookvale's prompt as 12,060. The shape is what matters. A prompt 105 times longer (12,327 / 117) made the first token wait about 171 times longer (7,540 / 44). Each output token got only 1.7 times slower (12.6 / 7.5). The wait grows with the prompt; the stream barely notices it.

One difference from the H100 numbers above: a laptop GPU has far less arithmetic for its memory speed, so here the long prefill takes about twice as long as all 300 output tokens together (7,540 ms against 300 x 12.6 ms = 3,780 ms). Lesson 4 shows why arithmetic sets the speed of prefill.

Streaming does not make any of this faster. The total time is the same. It changes what the user sees during that time. OpenAI's latency guide calls streaming "The single most effective approach, as it cuts the waiting time to a second or less", and adds:

"There's a huge difference between waiting and watching progress happen"

That is why Cookvale streams. A person at the stove with wet hands wants the first words fast, even if the last ones take another two seconds.

Recap. Prefill is one large pass over the whole prompt, and the user waits for it. Decode is one small pass per output token, and the user watches it. Time to first token measures the first; time per output token measures the second.

Where the Call Lives While It Runs

The call runs on a GPU, a chip built to do a very large number of multiplications at the same time. It has its own memory, which this course calls GPU memory (on data center GPUs, a kind called HBM, high bandwidth memory). The H100 has 80 GB. Three things share it while Priya's call runs.

The weights come first. Every pass uses every weight, so they stay in GPU memory all the time. The open 8B model stores each parameter in bf16, a number format that uses two bytes:

Llama 3.1 8B weights
  8,030,261,248 parameters x 2 bytes = 16,060,522,496 bytes = 16.06 GB
 
H100 GPU memory                       80 GB
used by the weights                   16.06 GB
left for everything else              80 - 16.06 = 63.94 GB

Second, this call's own saved work. During prefill, the model computes some numbers for every prompt token that later passes need again. Instead of redoing them on every pass, it keeps them in GPU memory. This is the KV cache (K and V stand for keys and values, the two kinds of numbers it saves). It belongs to this one call, grows by one token's worth with every decode pass, and is freed when the call ends. Lesson 5 shows what is in it and how big it gets.

Third, other people's calls. Priya's call almost certainly does not have a GPU to itself. The same GPU runs many calls at once, each with its own KV cache in the same memory. How that works, and what it costs each user, is lesson 6.

Reading the Stream

With stream: true, the provider sends a series of small events back over one HTTP connection. Here is a real transcript from Anthropic's streaming documentation, for a reply of 15 output tokens:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-opus-5-5", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}}
event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}
event: ping
data: {"type": "ping"}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}}
event: content_block_stop
data: {"type": "content_block_stop", "index": 0}
event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence":null}, "usage": {"output_tokens": 15}}
event: message_stop
data: {"type": "message_stop"}

Read it in order, with prefill and decode in mind.

• message_start comes first and already carries "input_tokens": 25. The input count is known before any text arrives, because prefill has read the whole prompt before the first output token exists. For Priya's call it would say 12,060.
• content_block_start opens a block of text. ping keeps the connection alive.
• Each content_block_delta carries a small piece of text, from one or more decode passes: "Hello", then "!". In Priya's chat panel, each one adds a few more words.
• content_block_stop closes the text block.
• message_delta says why the call stopped ("end_turn": the model finished its answer) and gives "output_tokens": 15. The output count can only be final at the end, because the length of the answer is not decided until the last decode pass. Anthropic's docs note that these counts "are cumulative": totals so far, not additions.
• message_stop ends the stream.

OpenAI's Responses API follows the same order with other names: response.created first, with "usage": null; text in response.output_text.delta events; the full usage at the end, in response.completed.

Check today. On 2026-10-18, Anthropic's stream gave the input token count in message_start and the final output count in message_delta; OpenAI's Responses API gave the whole usage block only in response.completed. Providers add event types as they add features, and Anthropic's docs ask your code to handle unknown ones. Check the current event list before you parse a stream: https://platform.claude.com/docs/en/build-with-claude/streaming

Reading the Usage Block

The last thing Priya's code receives is the usage block, the record the bill is computed from. Here are its fields as OpenAI documents them, verbatim from the response.completed example in its prompt caching guide (all zeros in the docs):

"usage": {
 "input_tokens": 0,
 "output_tokens": 0,
 "output_tokens_details": {
 "reasoning_tokens": 0
 },
 "total_tokens": 0,
 "input_tokens_details": {
 "cached_tokens": 0,
 "cache_write_tokens": 0
 }
}

• input_tokens is every token read in prefill, after the template. For Priya's turn 1, 12,060.
• output_tokens is every token written in decode. For Priya's turn 1, 300.
• reasoning_tokens is the part of the output the model wrote before the answer and did not show. These are often called thinking tokens: output tokens written before the answer. They are decode passes like any other and are billed as output. Cookvale's planning numbers assume none.
• total_tokens is input plus output: 12,360 for Priya's call.
• cached_tokens and cache_write_tokens belong to prompt caching, lesson 7. Until then they are zero.

Anthropic's usage block uses some different field names and splits the input differently once caching is on, but input_tokens and output_tokens mean the same thing there.

Recap. While the call runs, GPU memory holds the weights, this call's KV cache and other people's calls. The stream reports the input count first, because prefill is already done, and the output count last, because decode decides it.

Why Output Costs More, and Which Part of the Bill Is Bigger

Per Token, Output Is Far More Work

Take the H100 numbers again, per token:

time per input token    0.44 s / 12,060 tokens   = 0.0365 ms per input token
time per output token   1,600 ms / 300 tokens    = 5.33 ms per output token
 
output / input          5.33 / 0.0365            = about 146

For one user, one output token takes about 146 times the GPU time of one input token. Input tokens share one big pass; every output token needs a pass of its own.

So why is the price only five times higher, not 146 times? Because a provider runs many users' decode passes together, which recovers most of that cost. Providers do not publish how they set prices; this is the course's reading of the mechanism, and lesson 4 and lesson 6 work it through.

Per Call, Input Is the Bigger Bill

Prediction: output costs five times more per token. Which line of Priya's bill is bigger?

The course's price card comes from two current mid-size models, Claude Sonnet 5.5 and GPT-6.1 Sol, which both list exactly these prices (checked 2026-10-18):

mid-size model, per million tokens
  input     $2.00
  output    $10.00   (5 x input)

Priya's turn 1:

input    12,060 tokens x $2.00  / 1,000,000 = $0.02412
output      300 tokens x $10.00 / 1,000,000 = $0.00300
                                              --------
total                                         $0.02712
 
input / output cost      $0.02412 / $0.00300 = about 8
input / output tokens    12,060 / 300        = about 40

The input line is about eight times the output line. Each output token costs five times as much, but there are about forty times as many input tokens, and forty divided by five is eight. The dollar amounts will change; the ratio has been steadier. On 2026-10-18, every current mid-size and frontier model from the three largest providers priced output at exactly five times input.

Both facts hold at once. Per token, output is the expensive part, in GPU time and in price. Per call, input is the expensive part for Cookvale, because the prompt is long and the answer is short. "Output costs five times more" is true, and it points at the wrong line of this bill. Lesson 2 turns this into Cookvale's daily bill, and lesson 9 designs against it.

Recap. Each output token needs its own pass, so per token it is far more work than an input token, and it is priced higher. But Cookvale's prompt is forty times longer than its answer, so the input line of the bill is eight times the output line.

The Puzzle, Answered

Where did twelve thousand tokens come from? From the whole prompt, not the question: tools 2,000, system prompt 6,000, recipe 4,000 and question 60, rendered into one string of 12,060 tokens. Every later turn reads all of it again.

Why the pause, then the rush? Prefill read all 12,060 tokens before the first word could exist. Then each decode pass added one token, quickly. On the open model on an H100: about 0.44 s, then about 1.6 s.

Why is output priced higher, and input still bigger? Each output token needs a pass of its own, so per token it takes far more GPU time. But the call had forty times more input than output, so the input line ($0.02412) is eight times the output line ($0.003).

The Takeaway

1. The provider receives JSON; the model reads one flat string. A chat template wraps each message in marker tokens and writes the tools in as text.
2. Nothing is kept between calls. Every turn sends the whole conversation again: 12,060 input tokens on turn 1, 360 more on each later turn.
3. One pass ends with a score for every token in the vocabulary; the sampler picks one, it is added, and the model runs again. A 300-token answer is 300 passes.
4. Prefill is the first pass, over the whole prompt, felt as time to first token. Decode is every pass after it, one token each, felt as time per output token.
5. While the call runs, GPU memory holds the weights, this call's KV cache and other people's calls. The stream reports input first and output last.
6. Per token, output is far more work and priced five times higher. Per call, Cookvale's input is forty times its output, so input is eight times output on the bill.

The one sentence to keep: a model call is one big pass over your whole prompt, then one small pass for every output token, and every cost and delay you will meet comes from those two parts.

An open question to carry forward: if every output token needs a full pass over the model, how can a provider afford to serve thousands of people at once? Lesson 6 answers it.

References

Model files and templates (Hugging Face, checked 2026-10-18):

• Qwen/Qwen3-0.6B: the tokenizer and chat template used in the code above; vocab_size 151,936 and eos_token_id 151645 in config.json
• Qwen/Qwen3-8B tokenizer_config.json: the Qwen3 chat template whose text the rendered string above follows, including the # Tools block and the empty thinking block for enable_thinking=False
• Llama 3.1 8B Instruct, ungated mirror: the chat template with the "Cutting Knowledge Date" and "Today Date: 26 Jul 2024" header, and config.json (8,030,261,248 parameters, 16.06 GB in bf16). The official repository, meta-llama/Llama-3.1-8B-Instruct, is gated.
• The Llama 3.1 20 of 51 token measurement: rendered with jinja2 and cross-checked with transformers apply_chat_template on an Apple M1 Max, 2026-10-18 (transformers 5.20.1, tokenizers 0.23.3)

Provider documentation (checked 2026-10-18):

• Messages API reference, usage field (Anthropic): "the token counts in usage will not match one-to-one with the exact visible content of an API request or response"
• Context windows (Anthropic): "Input phase: Contains all previous conversation history plus the current user message"
• Streaming (Anthropic): the event transcript, and "The token counts shown in the usage field of the message_delta event are cumulative"
• Streaming responses (OpenAI) and Responses streaming events (OpenAI): response.created, response.output_text.delta, response.completed
• Prompt caching (OpenAI): the usage block from the response.completed example
• Latency optimization (OpenAI): streaming "cuts the waiting time to a second or less", "There's a huge difference between waiting and watching progress happen"
• Pricing (Anthropic) and Pricing (OpenAI): the mid-size price card, $2.00 input and $10.00 output per million tokens

Papers and engineering writing:

• Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023, arXiv:2211.17192: "decoding K tokens takes K serial runs of the model."
• Agarwal et al. (Databricks), "LLM Inference Performance Engineering: Best Practices", 2023-10-12: "Output length dominates overall response latency"

Hardware:

• NVIDIA H100: 80 GB of HBM3 and 989 teraflops of dense bf16 arithmetic on the SXM model (the page's 1,979 figure is marked "With sparsity")

Lab runs: the chat template script with Python 3.12, transformers 5.20.1 and tokenizers 0.23.3; the timing run with mlx 0.32.4 and mlx-lm 0.32.2. Both use Qwen/Qwen3-0.6B (bf16 weights) on an Apple M1 Max.