Loading video…
Sign in to react
One callSolid

The pause, the rush and the bill

Back in Cookvale, a user types eight words to Priya's cooking assistant, and her code sends 12,060 input tokens: one long string that the model reads in full, adding one new token with each pass, so a 300-token answer takes 300 passes. On the price list, each output token costs five times as much as an input token. When the call runs, the answer box stays empty for almost a second, and then the words rush in.

That leaves two questions:

  1. Why was there a pause, and then a rush of words?
  2. Why is output priced higher per token, and why is input still the bigger part of the bill?

The Pause, Then the Rush

Back to what Priya saw: almost a second of nothing, then a quick stream. The loop explains both.

Pass 1 is not like the others. When it starts, the model has never seen this prompt, so it must process all 12,060 tokens before it can score anything. Because all of them are known in advance, the GPU handles them together, in one large run. That run takes time, and the user sees nothing until it ends with the first output token.

Every later pass has only one new token. Each is small and quick, and as each one ends, one more token can go to the user.

These two parts have names:

  • Prefill is the first pass, over the whole prompt at once. It ends when the first output token exists.
  • Decode is every pass after it, each adding one new token.

The user feels them as two numbers on a clock:

  • Time to first token (often written TTFT) runs from sending the request until the first token arrives. For a long prompt, prefill is most of it. On a real provider it also includes the network and any time the request waits in a queue for a GPU; providers do not publish how that time splits.
  • Time per output token (TPOT) is the time between one streamed token and the next: the length of one decode pass. "Stream speed", in tokens per second, is a fine plain name for it.

The pause is prefill. The rush is decode.

How big are the two parts? Providers do not publish their model sizes or hardware, so we use an open model of known size on a known GPU: Llama 3.1 8B (8,030,261,248 parameters) on one NVIDIA H100, a common data center GPU. Counting the work, forcing the shape derives the amount of arithmetic and prefill and decode the time of one decode pass. Here we use their results:

Text
Llama 3.1 8B on one H100 (open model, arithmetic only)
 
prefill: one pass over 12,060 tokens
  arithmetic needed       about 219 trillion operations
  H100 peak speed         989 trillion operations per second
  time at full speed      219 / 989 = 0.22 s
  time at a realistic     about half of full speed
    half speed            0.22 s x 2 = about 0.44 s
 
decode: 300 passes of one token each
  one pass                about 5.3 ms
  300 passes              300 x 5.3 ms = 1,590 ms = about 1.6 s
 
decode time / prefill time    1.6 s / 0.44 s = about 4

The prompt is about 40 times longer than the answer (12,060 / 300 = 40.2). Yet the 300 output tokens take about four times as long as the 12,060 input tokens. Priya's "almost a second" comes from a larger model on hardware she cannot see, plus the network and the queue, but the shape is the same: a wait for the whole prompt, then a steady stream. Databricks's engineers state the practical rule in their guide to serving models:

"Output length dominates overall response latency"

The course measured both parts on a laptop, with Qwen3-0.6B, a small open model, and three prompts shaped like Cookvale's: a short system prompt with a recipe summary, a short system prompt with the full recipe, and the full turn-1 prompt with tools, system prompt, recipe and notes. Each ends in the butter question and gets 300 new tokens, always taking the highest-scoring token. The run uses MLX, Apple's machine learning library for its own chips. A model this small on a laptop is limited partly by the fixed cost of starting each pass, so read the numbers as a picture of the two phases, not of a data center. Prefill and decode shows the script and explains what limits each phase.

Run on an Apple M1 Max laptop:

Text
engine=mlx-lm model=Qwen/Qwen3-0.6B greedy, 300 new tokens (end-of-turn tokens banned so every run writes 300)
prompt | prompt tok |  TTFT ms | ms per output token | output tok/s | tokens out | peak GB
 small |        117 |       44 |                 7.5 |        134.7 |        300 |    1.41
medium |      1,010 |      197 |                 7.6 |        132.4 |        300 |    1.91
 large |     12,327 |     7540 |                12.6 |         79.5 |        300 |    3.19

The large prompt is 12,327 tokens here because Qwen3's tokenizer cuts the same text differently from the provider's, which bills Cookvale's prompt as 12,060. The shape is what matters. A prompt 105 times longer (12,327 / 117) made the first token wait about 171 times longer (7,540 / 44). Each output token got only 1.7 times slower (12.6 / 7.5). The wait grows with the prompt; the stream barely notices it.

One difference from the H100 numbers above: a laptop GPU has far less arithmetic for its memory speed, so here the long prefill takes about twice as long as all 300 output tokens together (7,540 ms against 300 x 12.6 ms = 3,780 ms). Why output costs more shows why arithmetic sets the speed of prefill.

Streaming does not make any of this faster. The total time is the same. It changes what the user sees during that time. OpenAI's latency guide calls streaming "The single most effective approach, as it cuts the waiting time to a second or less", and adds:

"There's a huge difference between waiting and watching progress happen"

That is why Cookvale streams. A person at the stove with wet hands wants the first words fast, even if the last ones take another two seconds.

Recap. Prefill is one large pass over the whole prompt, and the user waits for it. Decode is one small pass per output token, and the user watches it. Time to first token measures the first; time per output token measures the second.

Where the Call Lives While It Runs

The call runs on a GPU, a chip built to do a very large number of multiplications at the same time. It has its own memory, which this course calls GPU memory (on data center GPUs, a kind called HBM, high bandwidth memory). The H100 has 80 GB. Three things share it while Priya's call runs.

The weights come first. Every pass uses every weight, so they stay in GPU memory all the time. The open 8B model stores each parameter in bf16, a number format that uses two bytes:

Text
Llama 3.1 8B weights
  8,030,261,248 parameters x 2 bytes = 16,060,522,496 bytes = 16.06 GB
 
H100 GPU memory                       80 GB
used by the weights                   16.06 GB
left for everything else              80 - 16.06 = 63.94 GB

Second, this call's own saved work. During prefill, the model computes some numbers for every prompt token that later passes need again. Instead of redoing them on every pass, it keeps them in GPU memory. This is the KV cache (K and V stand for keys and values, the two kinds of numbers it saves). It belongs to this one call, grows by one token's worth with every decode pass, and is freed when the call ends. The KV cache shows what is in it and how big it gets.

Third, other people's calls. Priya's call almost certainly does not have a GPU to itself. The same GPU runs many calls at once, each with its own KV cache in the same memory. How that works, and what it costs each user, is the subject of batching many users.

Reading the Stream

With stream: true, the provider sends a series of small events back over one HTTP connection. Here is a real transcript from Anthropic's streaming documentation, for a reply of 15 output tokens:

Text
event: message_start
data: {"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-opus-5-5", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}}
event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}
event: ping
data: {"type": "ping"}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}}
event: content_block_stop
data: {"type": "content_block_stop", "index": 0}
event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence":null}, "usage": {"output_tokens": 15}}
event: message_stop
data: {"type": "message_stop"}

Read it in order, with prefill and decode in mind.

  • message_start comes first and already carries "input_tokens": 25. The input count is known before any text arrives, because prefill has read the whole prompt before the first output token exists. For Priya's call it would say 12,060.
  • content_block_start opens a block of text. ping keeps the connection alive.
  • Each content_block_delta carries a small piece of text, from one or more decode passes: "Hello", then "!". In Priya's chat panel, each one adds a few more words.
  • content_block_stop closes the text block.
  • message_delta says why the call stopped ("end_turn": the model finished its answer) and gives "output_tokens": 15. The output count can only be final at the end, because the length of the answer is not decided until the last decode pass. Anthropic's docs note that these counts "are cumulative": totals so far, not additions.
  • message_stop ends the stream.

OpenAI's Responses API follows the same order with other names: response.created first, with "usage": null; text in response.output_text.delta events; the full usage at the end, in response.completed.

Check today. On 2026-10-18, Anthropic's stream gave the input token count in message_start and the final output count in message_delta; OpenAI's Responses API gave the whole usage block only in response.completed. Providers add event types as they add features, and Anthropic's docs ask your code to handle unknown ones. Check the current event list before you parse a stream: https://platform.claude.com/docs/en/build-with-claude/streaming

Reading the Usage Block

The last thing Priya's code receives is the usage block, the record the bill is computed from. Here are its fields as OpenAI documents them, verbatim from the response.completed example in its prompt caching guide (all zeros in the docs):

JSON
"usage": {
 "input_tokens": 0,
 "output_tokens": 0,
 "output_tokens_details": {
 "reasoning_tokens": 0
 },
 "total_tokens": 0,
 "input_tokens_details": {
 "cached_tokens": 0,
 "cache_write_tokens": 0
 }
}
  • input_tokens is every token read in prefill, after the template. For Priya's turn 1, 12,060.
  • output_tokens is every token written in decode. For Priya's turn 1, 300.
  • reasoning_tokens is the part of the output the model wrote before the answer and did not show. These are often called thinking tokens: output tokens written before the answer. They are decode passes like any other and are billed as output. Cookvale's planning numbers assume none.
  • total_tokens is input plus output: 12,360 for Priya's call.
  • cached_tokens and cache_write_tokens belong to prompt caching. Until then they are zero.

Anthropic's usage block uses some different field names and splits the input differently once caching is on, but input_tokens and output_tokens mean the same thing there.

Recap. While the call runs, GPU memory holds the weights, this call's KV cache and other people's calls. The stream reports the input count first, because prefill is already done, and the output count last, because decode decides it.

Why Output Costs More, and Which Part of the Bill Is Bigger

Per Token, Output Is Far More Work

Take the H100 numbers again, per token:

Text
time per input token    0.44 s / 12,060 tokens   = 0.0365 ms per input token
time per output token   1,600 ms / 300 tokens    = 5.33 ms per output token
 
output / input          5.33 / 0.0365            = about 146

For one user, one output token takes about 146 times the GPU time of one input token. Input tokens share one big pass; every output token needs a pass of its own.

So why is the price only five times higher, not 146 times? Because a provider runs many users' decode passes together, which recovers most of that cost. Providers do not publish how they set prices; this is the course's reading of the mechanism, and why output costs more and batching many users work it through.

Per Call, Input Is the Bigger Bill

Prediction: output costs five times more per token. Which line of Priya's bill is bigger?

The course's price card comes from two current mid-size models, Claude Sonnet 5.5 and GPT-6.1 Sol, which both list exactly these prices (checked 2026-10-18):

Text
mid-size model, per million tokens
  input     $2.00
  output    $10.00   (5 x input)

Priya's turn 1:

Text
input    12,060 tokens x $2.00  / 1,000,000 = $0.02412
output      300 tokens x $10.00 / 1,000,000 = $0.00300
                                              --------
total                                         $0.02712
 
input / output cost      $0.02412 / $0.00300 = about 8
input / output tokens    12,060 / 300        = about 40

The input line is about eight times the output line. Each output token costs five times as much, but there are about forty times as many input tokens, and forty divided by five is eight. The dollar amounts will change; the ratio has been steadier. On 2026-10-18, every current mid-size and frontier model from the three largest providers priced output at exactly five times input.

Both facts hold at once. Per token, output is the expensive part, in GPU time and in price. Per call, input is the expensive part for Cookvale, because the prompt is long and the answer is short. "Output costs five times more" is true, and it points at the wrong line of this bill. Counting every token turns this into Cookvale's daily bill, and where the money goes designs against it.

Recap. Each output token needs its own pass, so per token it is far more work than an input token, and it is priced higher. But Cookvale's prompt is forty times longer than its answer, so the input line of the bill is eight times the output line.

The Puzzle, Answered

Why the pause, then the rush? Prefill read all 12,060 tokens before the first word could exist. Then each decode pass added one token, quickly. On the open model on an H100: about 0.44 s, then about 1.6 s.

Why is output priced higher, and input still bigger? Each output token needs a pass of its own, so per token it takes far more GPU time. But the call had forty times more input than output, so the input line ($0.02412) is eight times the output line ($0.003).

The Takeaway

  1. Prefill is the first pass, over the whole prompt, felt as time to first token. Decode is every pass after it, one token each, felt as time per output token.
  2. While the call runs, GPU memory holds the weights, this call's KV cache and other people's calls. The stream reports input first and output last.
  3. Per token, output is far more work and priced five times higher. Per call, Cookvale's input is forty times its output, so input is eight times output on the bill.

The one sentence to keep: a model call is one big pass over your whole prompt, then one small pass for every output token, and every cost and delay you will meet comes from those two parts.

An open question to carry forward: if every output token needs a full pass over the model, how can a provider afford to serve thousands of people at once? Batching many users answers it.

References

Model files (Hugging Face, checked 2026-10-18):

Provider documentation (checked 2026-10-18):

Papers and engineering writing:

Hardware:

  • NVIDIA H100: 80 GB of HBM3 and 989 teraflops of dense bf16 arithmetic on the SXM model (the page's 1,979 figure is marked "With sparsity")

Lab runs: the timing run with mlx 0.32.4 and mlx-lm 0.32.2, using Qwen/Qwen3-0.6B (bf16 weights) on an Apple M1 Max.