Counting every token

Priya's usage log for Cookvale's first real call says 12,060 input tokens. Before launch, a colleague estimated that number from a word count, and it was right only because two errors cancelled: the rule of thumb overcounted the prose by 2,000 tokens, and nobody counted the 2,000 tokens of tool definitions. The user typed about 60 of those tokens. Now we account for the other twelve thousand, one part at a time.

What Else Is Inside the 12,060

Priya's own text is the system prompt, the recipe and the question. But the model reads more than her text.

The Chat Template

The first lesson showed that the provider turns the list of messages into one long string before tokenizing it. The format that does this is the chat template: it wraps each message in marker tokens, special tokens that say where a turn starts, who is speaking and where the turn ends. The model was trained on text in exactly this format.

Open models publish their template, so you can see it. Here is the real Qwen3 render of a two-message list (a one-line system prompt and one question), from the research for this course:

<|im_start|>system
You are a helpful support agent.<|im_end|>
<|im_start|>user
Where is my order A-1042?<|im_end|>
<|im_start|>assistant

<|im_start|> and <|im_end|> are single tokens (IDs 151644 and 151645). The words "system", "user" and "assistant" and the line breaks are tokens too. The two messages' text alone is 18 tokens. The rendered string is 31 tokens. The template added 13 tokens for two short messages, and you never wrote any of them.

Cookvale's own two messages from the first lesson go through the same template, with thinking turned off. This script prints the token IDs, then each ID turned back into its text:

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
 
messages = [
    {"role": "system", "content": "You are Cookvale's cooking assistant. Answer in one or two short sentences."},
    {"role": "user", "content": "Can I use butter instead of oil?"},
]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
 
ids = tok(text)["input_ids"]
print(ids)
print([tok.decode([i]) for i in ids])

Run on an Apple M1 Max laptop:

[151644, 8948, 198, 2610, 525, 12514, 77729, 594, 17233, 17847, 13, 21806, 304, 825, 476, 1378, 2805, 22870, 13, 151645, 198, 151644, 872, 198, 6713, 358, 990, 14100, 4518, 315, 5590, 30, 151645, 198, 151644, 77091, 198, 151667, 271, 151668, 271]
['<|im_start|>', 'system', '\n', 'You', ' are', ' Cook', 'vale', "'s", ' cooking', ' assistant', '.', ' Answer', ' in', ' one', ' or', ' two', ' short', ' sentences', '.', '<|im_end|>', '\n', '<|im_start|>', 'user', '\n', 'Can', ' I', ' use', ' butter', ' instead', ' of', ' oil', '?', '<|im_end|>', '\n', '<|im_start|>', 'assistant', '\n', '<think>', '\n\n', '</think>', '\n\n']

The two messages' text alone is 24 tokens (the first lesson's script counted it); the rendered string is 41, the length of the list above. Thirteen of the 17 extra tokens are the same markers as above: <|im_start|>, a role word and a line break before each message, <|im_end|> and a line break after it. The other four are the empty thinking block that enable_thinking=False adds: <think> (151667), a blank line (\n\n, 271), </think> (151668) and another blank line. Notice also that "Cookvale" is two tokens, Cook and vale. A made-up name costs more than a common word.

Closed providers do the same thing on their servers. OpenAI's token counting guide says the count "includes formatting tokens used to represent request structure, such as message roles and boundaries. These tokens might not appear in the text or fields you tokenize locally."

Output has hidden tokens too. OpenAI says some models "generate tokens used to format or delimit response channels, tool calls, and other message structure", so "the reported output or completion token count can be higher than the number of visible tokens".

Tool Definitions Are Text

Predict before reading on. Take a short message and add one small tool definition: a name, a one-line description and one argument. How many tokens does it add?

To answer, look at what a tool becomes. The model has no special input for tools. The template writes each tool definition into the prompt as text, with instructions on how to call it. Here is part of the real Qwen3 render with one tool, from the research for this course:

# Tools
 
You may call one or more functions to assist with the user query.
 
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_order", "description": "Look up an order", "parameters": {"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}}}
</tools>
 
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
...

Every one of those tokens is input, on every call. Anthropic's token counting page has a clean measurement. A system prompt ("You are a scientist") and the message "Hello, Claude" count as:

{ "input_tokens": 14 }

One small get_weather tool (a name, a one-line description, one location argument) and a one-line weather question count as:

{ "input_tokens": 403 }

Anthropic's pricing page explains most of the gap: "When you use tools, the API also automatically includes a special system prompt for the model that enables tool use." Its table lists that hidden system prompt at 286 tokens on its current models (with tool_choice set to auto or none).

with one tool                                403 tokens
hidden tool-use system prompt              - 286 tokens
the tool's JSON definition + the question  = 117 tokens
 
without tools (different short message)       14 tokens

So one small tool took a short request from tens of tokens to about 400. Most of the cost was instructions you never see.

Cookvale, Band by Band

Now every token of Cookvale's turn 1 can be accounted for. The order is the order providers render: tools, then the system prompt, then the recipe, then the question.

tool definitions   2,000 tokens   8 tools (find_recipes, get_recipe, scale_recipe,
                                  convert_units, substitute_ingredient, set_timer,
                                  add_to_shopping_list, get_pantry), JSON schemas,
                                  plus the provider's tool instructions
system prompt      6,000 tokens   voice and house rules, food safety, units,
                                  six worked example exchanges
retrieved recipe   4,000 tokens   the recipe being cooked (about 900 tokens)
                                  plus three technique notes the app attaches
user question         60 tokens   average across all five languages
 
total input        2,000 + 6,000 + 4,000 + 60 = 12,060 tokens

The user typed 60 of those tokens. The other 12,000 are Cookvale's own text: the same 8,000 tokens of tools and system prompt for every user, plus the recipe. Every tool Priya adds costs tokens on every call, whether the user needs that tool or not.

Format Changes the Count

Here is the main prediction of this lesson. Take some time with it.

Cookvale's app stores each recipe as JSON. The app can paste the recipe into the prompt as pretty JSON (indented, one field per line), as minified JSON (no spaces), or as plain text. Same information. How much more does the pretty JSON cost than the plain text?

Think about what JSON is made of. Every field name appears again in every item: "quantity" is written once per ingredient. Every name sits in quotes. Every value is followed by a comma. Braces and brackets open and close each object. Pretty JSON adds runs of spaces at the start of every line, and a run of spaces is its own token. None of those characters are frequent English words, so the merge rules cut them into small pieces.

Here is a real JSON fragment as o200k_base tokens, from the research for this course:

text     {"user_id": 4521, "status": "shipped"}
 
tokens   [{"] [user] [_id] [":] [ ] [452] [1] [,] [ "] [status] [":] [ "] [sh] [ipped] ["}]
         15 tokens; [ ] is a single space, [ "] is a space and a quote

{" and ": are single tokens. The number 4521 is two. The common word "status" is one, but "shipped" is two. The structure costs about as many tokens as the data.

The research measured the same 50 order records in three formats (fields order_id, customer_email, status, total_cents, created_at):

pretty JSON / CSV           = 3,187 / 1,447 = 2.20
saving from minifying alone = 1 - 2,238 / 3,187 = 0.298, about 30 percent

Pretty JSON cost 2.2 times the same data as CSV. CSV says each field name once, in the header. JSON says it on every row. CSV is the plain-text form the research measured; the counts for a Cookvale recipe come from the lab:

The lab's script builds one Cookvale recipe (lemon herb chicken thighs with crispy potatoes: 13 ingredients and 9 steps) as a Python dictionary, then counts it three ways: json.dumps(recipe, indent=2) for pretty JSON, json.dumps(recipe, separators=(",", ":")) for minified JSON, and plain text with one line per ingredient and per step. Run on an Apple M1 Max laptop:

format         | chars | o200k tokens | vs plain | Qwen3 tokens
pretty JSON    |  1875 |          570 |    2.10x |          590
minified JSON  |  1345 |          359 |    1.32x |          377
plain text     |   940 |          272 |    1.00x |          294
 
first 24 o200k tokens of the pretty JSON: ['{\n', ' ', ' "', 'id', '":', ' "', 'rc', 'p', '_', '8', 'f', '21', 'c', '",\n', ' ', ' "', 'title', '":', ' "', 'L', 'emon', ' herb', ' chicken', ' thighs']

Pretty JSON cost 2.10 times the plain text (570 / 272). Minifying alone saved 37 percent (1 - 359 / 570). The first tokens show where the extra goes: on every line, a line break, two spaces of indent, a quote, the key, and a quote with a colon.

There is a trade-off: a model may read a flat format less reliably for deeply nested or very wide data. A recipe (a title, ingredients, steps) is neither, so plain text suits it. Tool definitions must stay JSON schemas, because providers require that format.

The Same Letters, Twice the Tokens

One more surprise, for text users type. Unicode can store many accented letters two ways. NFC stores "é" as one character; NFD stores it as "e" plus a separate accent mark. They look identical, but they are different bytes, so they become different tokens.

The research found a real case. The Vietnamese translation of the Universal Declaration of Human Rights was stored in NFD. On o200k_base it was 6,950 tokens. The same text converted to NFC was 3,093 tokens.

NFD / NFC = 6,950 / 3,093 = 2.25

Same visible text, 2.25 times the tokens. Qwen3's tokenizer converts text to NFC itself, so it counted 3,032 either way. tiktoken and Llama 3 do not convert.

The lab ran the same check on Cookvale text: the Spanish butter answer and two recipe titles, each in NFC and in NFD, counted on o200k_base and on Qwen3. Run on an Apple M1 Max laptop:

text               | form | chars | o200k tokens | Qwen3 tokens | looks identical
Spanish answer     | NFC  |   972 |          258 |          312 | True
Spanish answer     | NFD  |   991 |          284 |          312 | True
recipe title (fr)  | NFC  |    53 |           17 |           20 | True
recipe title (fr)  | NFD  |    60 |           24 |           20 | True
recipe title (vi)  | NFC  |    43 |           17 |           18 | True
recipe title (vi)  | NFD  |    56 |           30 |           18 | True
 
o200k pieces, NFC: Cr|ème| brû|lée| à| la| van|ille|,| café| et| noix| de| coco| râ|p|ée
o200k pieces, NFD: Cre|<U+0300>|me| bru|<U+0302>|le|<U+0301>|e| a<U+0300>| la| van|ille|,| cafe|<U+0301>| et| noix| de| coco| ra|<U+0302>|pe|<U+0301>|e
(<U+0300>, <U+0301>, <U+0302> are the combining accent marks that NFD stores as separate characters)

Every row looks identical on screen. On o200k_base the Spanish answer costs 10 percent more in NFD (284 / 258), and the Vietnamese title 76 percent more (30 / 17), because Vietnamese puts more accents on each word. The last two lines show why: in NFD each accent mark becomes a token of its own. The Qwen3 column does not move.

Text copied from some PDFs, from macOS file names and from some keyboards arrives as NFD. Converting user text to NFC before sending is one line of Python:

import unicodedata
 
clean = unicodedata.normalize("NFC", user_text)  # combine letter + accent into one character

Recap: the template, the tool instructions and the format of your data are all tokens. One small tool took a request from 14 to 403 tokens, and pretty JSON cost 2.2 times CSV for the same records.

What Counts Against the Context Window

The context window is the most tokens one call can hold. It is not only a limit on the prompt. It covers the prompt and the answer together. Anthropic's definition: "The "context window" refers to all the text a language model can reference when generating a response, including the response itself." And in more detail: "Everything in the request counts toward the context window: the system prompt, every message in messages (including tool results, images, and documents), and your tool definitions. The output Claude generates for the turn, including its extended thinking, counts too."

"Extended thinking" means thinking tokens: output tokens the model writes before the answer. They take space in the window and are billed as output, even when the user never sees them.

OpenAI's model pages make the sharing visible as two numbers. For its current mid-size model (checked 2026-10-18):

context window             1,050,000 tokens
maximum input tokens         922,000 tokens
room kept for the output   1,050,000 - 922,000 = 128,000 tokens

The 128,000 matches the model's maximum output. The window is one space, and input and output share it.

Every Turn Sends Everything Again

The first lesson showed that the API keeps nothing between calls. To continue a chat, the app sends the whole conversation again. Anthropic's docs describe a turn's input as: "Contains all previous conversation history plus the current user message".

A Cookvale chat has 4 turns on average. Each later turn adds the previous answer (300 tokens) and the new question (60 tokens):

turn 1 input = 2,000 + 6,000 + 4,000 + 60 = 12,060
turn 2 input = 12,060 + 300 + 60          = 12,420
turn 3 input = 12,420 + 360               = 12,780
turn 4 input = 12,780 + 360               = 13,140
 
input billed per chat  = 12,060 + 12,420 + 12,780 + 13,140 = 50,400 tokens
output billed per chat = 4 x 300                           =  1,200 tokens

Predict before reading on. Turn 4 sends 13,140 tokens. How many of them has the provider seen before, in this same chat?

All of them except the new question:

turn 4 input                          13,140 tokens
  new in turn 4 (the question)      -     60 tokens
  already sent or written before    = 13,080 tokens
  share already seen                = 13,080 / 13,140 = 99.5 percent

The 13,080 are turn 3's whole input plus the model's own turn 3 answer. Priya pays the input price for all 13,140, and the model reads them all again.

At the end of turn 4 the context holds 13,140 input tokens plus the 300-token answer: 13,440 tokens.

13,440 / 1,050,000 = 0.0128, about 1.3 percent of the window

Cookvale is nowhere near the context window. The limit that matters to Priya is the bill, and later lessons show it also matters to how much GPU memory each conversation needs. A request that does not fit is refused, not cut short. Anthropic: "If the input alone already exceeds the model's context window, the API returns a 400 invalid_request_error ("prompt is too long") on every model."

Counting Before You Send

The colleague's mistake had a simple cause: an estimate instead of a count. There are two ways to count.

The provider's counting endpoint. It takes the same request you would send and returns the number the provider will bill, template and tools included. Anthropic's (verbatim from its docs):

curl https://api.anthropic.com/v1/messages/count_tokens \
 -H "x-api-key: $ANTHROPIC_API_KEY" \
 -H "content-type: application/json" \
 -H "anthropic-version: 2023-06-01" \
 -d '{
 "model": "claude-opus-5-5",
 "system": "You are a scientist",
 "messages": [{
 "role": "user",
 "content": "Hello, Claude"
 }]
 }'

{ "input_tokens": 14 }

The model field matters: the endpoint counts with that model's tokenizer. Anthropic says counting is "free to use" and that "The token count is an estimate." OpenAI has the same kind of endpoint at POST /v1/responses/input_tokens. Both need an API key; this course does not require one.

A local tokenizer. tiktoken and the Hugging Face tokenizers library run on your laptop with no key. They are exact for plain text in the encoding you load, and they miss what the provider adds. OpenAI's guide: "Local tokenizers like tiktoken work for plain text, but they have limitations", including "Tools and schemas add tokens that are hard to count locally". For Anthropic's models there is no local tokenizer at all.

So use a local tokenizer to compare texts (English against Telugu, JSON against plain text) and the provider's endpoint to know the real number for a request.

Recap: input and output share the context window, thinking included. Every turn re-sends the whole chat, so Cookvale's turn 4 is 13,140 tokens, 99.5 percent of them already seen. Count with the provider's endpoint, compare with a local tokenizer.

What a Token Costs

Providers price tokens per million. Here is the price card for the mid-size model Cookvale uses:

mid-size model, per million tokens (checked 2026-10-18)
  input     $2.00
  output   $10.00   (5 times input)

Claude Sonnet 5.5 and GPT-6.1 Sol both list exactly these prices on 2026-10-18. Later lessons add the lines for cached input and batch.

One Call

Turn 1 of a chat:

input    12,060 tokens x $2  / 1,000,000 = $0.02412
output      300 tokens x $10 / 1,000,000 = $0.00300
total                                      $0.02712, about 2.7 cents
 
input cost / output cost = 0.02412 / 0.00300 = 8.04, about 8 times

Each output token costs five times an input token. But there are 40 times more input tokens (12,060 against 300), so the input line is 8 times bigger.

One Day

50,000 chats a day, each with 50,400 input tokens and 1,200 output tokens:

input per day    50,000 chats x 50,400 = 2,520,000,000 tokens
output per day   50,000 chats x  1,200 =    60,000,000 tokens
 
input    2,520,000,000 x $2  / 1,000,000 = $5,040
output      60,000,000 x $10 / 1,000,000 =   $600
total                                      $5,640 a day
 
input share       5,040 / 5,640 = 0.894, about 89 percent
per month         $5,640 x 30 = $169,200

Most of Cookvale's bill is tokens the provider has already seen: the same 8,000-token prefix for every user and the same chat re-sent on every turn.

What the Languages Add

The 300-token average answer is not a design choice. It is the language mix, weighted by each language's share of chats:

English    0.50 x 231 = 115.50
Spanish    0.20 x 282 =  56.40
Hindi      0.15 x 386 =  57.90
Japanese   0.10 x 407 =  40.70
Telugu     0.05 x 601 =  30.05
average answer        = 300.55, about 300 tokens

If every user wrote in English, the average answer would be 231 tokens. Compare the two:

output per day, real mix     200,000 calls x 300 = 60,000,000 tokens   $600
output per day, all English  200,000 calls x 231 = 46,200,000 tokens   $462
added by the language mix                                              $138 a day
 
one English answer   231 x $10 / 1,000,000 = $0.00231
one Telugu answer    601 x $10 / 1,000,000 = $0.00601

The language mix adds 30 percent to Cookvale's output tokens (300 / 231 = 1.30). It is a real line in the budget. The users chose their languages; the tokenizer decided what each language costs.

So the rule for this lesson and the one before it: you pay for tokens, and the language, the format and the tokenizer decide how many tokens a text becomes.

Check today. On 2026-10-18, OpenAI's tiktoken mapped its GPT-5 models to o200k_base and did not yet list the GPT-6 models; Anthropic's Claude 4.7 and later models used a newer tokenizer that makes about 30 percent more tokens than earlier Claude models, and Anthropic publishes no local tokenizer. Tokenizers change with model generations, so the language ratios in this lesson move too. Check the current model-to-encoding map: https://github.com/openai/tiktoken/blob/main/tiktoken/model.py

The Takeaway

1. The model reads more than your text: template markers (13 extra tokens for two short Qwen3 messages), tool instructions (one small tool took a request from 14 to 403 tokens) and the format of your data (pretty JSON cost 2.2 times CSV).
2. Input and output share the context window, thinking tokens included. Every turn re-sends the whole chat: 12,060, 12,420, 12,780, 13,140.
3. Count with the provider's counting endpoint for the real number; use a local tokenizer to compare texts.
4. Cookvale's turn 1 costs $0.02712, with input 8 times the output cost. A day costs $5,640, 89 percent of it input. The language mix adds 30 percent to the output tokens.

The one sentence to keep: you pay for tokens, not words, and the number of tokens a text becomes depends on its language, its format and the model's tokenizer.

An open question for later: every turn re-sends twelve thousand tokens the provider has already read. Does it have to read them all again? The KV cache shows what the model saves inside one call, and paged memory and prompt caching show what a provider can keep between calls.

References

Provider documentation, checked 2026-10-18:

• Token counting (Claude API docs): the count_tokens endpoint, the 14 and 403 examples, "estimate", "free to use", the newer tokenizer and the 30 percent note
• Pricing (Claude API docs): the tool use system prompt (286 tokens on current models), the tokenizer note, the price card
• Context windows (Claude API docs): the definition "including the response itself", what counts toward the window, "prompt is too long"
• Messages API reference (Claude API docs): the usage note on counts that do not match visible content
• Counting tokens (OpenAI API docs): POST /v1/responses/input_tokens, formatting tokens, the limits of local tokenizers, hidden output formatting tokens
• GPT-6.1 Sol model page (OpenAI API docs): 1,050,000-token context window, 922,000 maximum input tokens
• OpenAI Pricing: the price card

Tokenizers and tools:

• tiktoken 0.14.0 and its model-to-encoding map ("gpt-5": "o200k_base"), checked 2026-10-18
• Hugging Face tokenizers 0.23.3; Qwen3 tokenizer.json and chat template in tokenizer_config.json; Llama 3.1 tokenizer and template from the ungated mirror unsloth/Meta-Llama-3.1-8B-Instruct
• Unicode UDHR project: the parallel texts behind the language table, NFC-normalized for the table

Measurements: the JSON fragment, the 50-record format table, the Qwen3 and Llama 3.1 template renders and the Vietnamese NFC and NFD counts were made by the research for this course on an Apple M1 Max, 2026-10-18, with tiktoken 0.14.0, tokenizers 0.23.3 and transformers 5.20.1. The Cookvale template render, the recipe JSON counts and the Cookvale NFC and NFD counts come from the course's lab on the same laptop (tiktoken 0.14.0, transformers 5.20.1). Every other Cookvale figure is worked in this lesson.