Tokens: what you pay for

Cookvale is a recipe app. Its cooking assistant answers questions while a person cooks: can I swap this, how long does this take, is this done. Priya is the engineer who owns it. Last week Cookvale launched the assistant in five languages, and this morning Priya is looking at her usage dashboard.

Two rows on it make no sense to her. A user cooking in English asked whether the rice was done, and got an answer of 231 output tokens. A user cooking in Telugu asked the same question about the same recipe, and got the same advice, in Telugu. That answer was 601 output tokens. Same meaning, same model, same price per token. The Telugu answer cost 2.6 times as much.

Telugu answer / English answer = 601 / 231 = 2.60

There is a second puzzle on her desk. Before launch, a colleague estimated the size of the prompt: "our system prompt and a typical recipe come to about 9,000 words, so the prompt is about 12,000 tokens." The first real usage log said 12,060 input tokens. The estimate was right. But, as you will see in this lesson, it was right for the wrong reason, and the next estimate made the same way will be wrong.

Both puzzles have the same root. Priya pays for tokens, not words. So what is a token, and who decides how many tokens a piece of text becomes?

By the end of this lesson you will be able to explain how a tokenizer is built, predict which texts cost more, account for every one of the 12,060 tokens in a Cookvale call, and compute what a day of Cookvale costs.

What a Token Is

A token is one piece of text that the model reads or writes. The model never sees letters or words. It sees a list of integers, one per token. Each integer is a token ID. The program that cuts text into tokens, and turns tokens back into text, is the tokenizer. The full list of tokens a tokenizer knows is its vocabulary. Each model family has its own tokenizer: o200k_base has about 200,000 tokens in its vocabulary, Llama 3 has 128,000, and Qwen3 has 151,643 regular tokens.

Lesson 1 showed that a model call starts by turning the prompt into token IDs and ends by turning the output IDs back into text. This lesson opens that step.

How a Tokenizer Is Built

Almost every current tokenizer is built with byte pair encoding (BPE). It is a short procedure:

1. Start with a very large collection of training text.
2. Write every piece of text as single symbols.
3. Count every pair of neighbouring symbols across the whole collection.
4. Take the most frequent pair and make it one new symbol. Write that rule down.
5. Repeat from step 3, tens of thousands of times.

The list of rules, in order, is the tokenizer. To tokenize new text later, the program applies the same rules in the same order.

Here is the procedure on a tiny collection: the word "stir" three times, "stew" two times and "salt" once. Words are counted separately, the way real tokenizers first split text at spaces.

start      s t i r (x3)    s t e w (x2)    s a l t (x1)
 
round 1    pair counts: s t = 3 + 2 = 5, t i = 3, i r = 3, t e = 2, e w = 2,
           s a = 1, a l = 1, l t = 1
           most frequent: s t (5)            new symbol "st"
round 2    st i = 3, i r = 3 (a tie; the first one found wins)
                                             new symbol "sti"
round 3    sti r = 3                         new symbol "stir"
round 4    st e = 2, e w = 2 (a tie again)   new symbol "ste"
round 5    ste w = 2                         new symbol "stew"
 
after 5 rounds
  "stir"  -> 1 token    (frequent)
  "stew"  -> 1 token    (frequent)
  "salt"  -> 4 tokens   s | a | l | t   (seen once, never merged)

That last line is the whole idea. Strings that were frequent in the training text become one token. Rare strings stay in pieces. A real tokenizer runs the same loop with a much bigger collection and many more rounds. The Llama 3 tokenizer file holds 280,147 merge rules.

Two details make real tokenizers work on any text.

First, the starting symbols are bytes, not letters. Every text on a computer is stored as bytes, usually in the UTF-8 format, so a byte-level tokenizer can read any language, any emoji and any typo. The GPT-2 paper put it this way: "a byte-level version of BPE only requires a base vocabulary of size 256." There is never an unknown character. Notice what this means for different languages. In UTF-8, an English letter is 1 byte. A Spanish "ñ" is 2 bytes. A Hindi letter, a Telugu letter and a Japanese character are each 3 bytes. Before a single merge, a Telugu word already starts as about three times more symbols than an English word with the same number of letters.

Second, before merging, the tokenizer splits text into chunks with a fixed pattern (a regular expression) and never merges across those chunks. The GPT-2 authors explain why: without it, BPE learned "many versions of common words like dog since they occur in many variations such as dog. dog! dog?", so they "prevent BPE from merging across character categories". The pattern keeps a leading space attached to the word that follows it. On o200k_base, " stir" with a space is one token, and "stir" without one is two: st ir.

Strawberry, With and Without a Space

That last detail explains a failure many people have seen: models are bad at counting the letters in a word. Run this on any laptop:

import tiktoken  # tiktoken 0.14.0
 
enc = tiktoken.get_encoding("o200k_base")
 
for text in ["strawberry", " strawberry"]:
    ids = enc.encode(text)                      # text -> list of token IDs
    pieces = [enc.decode([i]) for i in ids]     # each ID back to its text
    print(repr(text), len(ids), pieces)

get_encoding loads the o200k_base rules. encode applies them and returns token IDs. The next line turns each ID back into its text, so you can see the cuts. The research for this course measured these splits with tiktoken 0.14.0 (2026-10-18):

In the middle of a sentence, "strawberry" follows a space, so the model receives one integer, with no letters to count. At the start of a line it receives three different integers. Neither input shows the letter "r". That is why spelling, counting letters and reversing strings are weak spots: the model works on IDs, not characters.

Numbers have their own rules. o200k_base and Llama 3 cut digits into groups of up to three: "1234567" becomes 123 456 7. Qwen3 cuts every digit alone, so the same number is 7 tokens. A recipe full of grams and temperatures costs a little more on Qwen3 for that reason.

You can see all of this live in a browser: Tiktokenizer colours each token of any text you paste and shows its ID.

Recap: a tokenizer is a list of merge rules learned from training text. Frequent strings become one token, rare strings stay in pieces, and the pieces are what you pay for.

Why Telugu Costs 2.6 Times English

Before reading on, predict. A Cookvale answer is translated into Telugu with exactly the same meaning. Will it be more tokens, fewer, or the same?

The procedure you just saw gives the answer. Merge rules come from the training text. The more often a string appeared in that text, the earlier it got merged into one token. So a tokenizer is cheap for the languages that filled its training text and expensive for the rest. English words like " the", " recipe" and " minutes" became single tokens early. Telugu words appeared far less often, so most of them were never merged, and each one stays as several pieces. Each piece starts from 3-byte letters.

The Measured Table

The research for this course counted the same text in many languages: the Universal Declaration of Human Rights, which exists in parallel translations. The ratio is the token count for a language divided by the token count for English, with the same tokenizer. Measured on 2026-10-18 on an Apple M1 Max with tiktoken 0.14.0 and the Qwen3 and Llama 3 tokenizer.json files:

Cookvale's answer sizes come from the o200k_base column. Start from the English answer, 231 tokens, and multiply by each ratio:

English    231 x 1.00 = 231 tokens
Spanish    231 x 1.22 = 281.8, about 282 tokens
Hindi      231 x 1.67 = 385.8, about 386 tokens
Japanese   231 x 1.76 = 406.6, about 407 tokens
Telugu     231 x 2.60 = 600.6, about 601 tokens

That is Priya's dashboard: 231 and 601.

A published parallel text is the fair way to compare languages, but your own text can land somewhere else. The lab for this course counted one real Cookvale answer, the reply to "Can I use butter instead of oil?" (164 English words), in all five languages on all four tokenizers. Each file holds one translation of the same answer:

import tiktoken                          # tiktoken 0.14.0
from transformers import AutoTokenizer   # transformers 5.20.1
 
o200k = tiktoken.get_encoding("o200k_base")
cl100k = tiktoken.get_encoding("cl100k_base")
qwen = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
llama = AutoTokenizer.from_pretrained("unsloth/Llama-3.2-1B-Instruct")  # the Llama 3 tokenizer
 
count = {
    "o200k_base": lambda t: len(o200k.encode(t)),
    "cl100k_base": lambda t: len(cl100k.encode(t)),
    "Qwen3": lambda t: len(qwen.encode(t, add_special_tokens=False)),
    "Llama 3": lambda t: len(llama.encode(t, add_special_tokens=False)),
}
 
langs = ["en", "es", "hi", "ja", "te"]
texts = {}
for lang in langs:
    with open(f"answer_{lang}.txt", encoding="utf-8") as f:
        texts[lang] = f.read().strip()
 
print("     " + "".join(f"{name:>16}" for name in count))
for lang in langs:
    cells = ""
    for name, n in count.items():
        tokens = n(texts[lang])
        cells += f"{tokens:>8} ({tokens / n(texts['en']):.2f}x)"
    print(f"{lang:5}" + cells)

count maps each tokenizer's name to a small function that returns a token count; Qwen3 and Llama 3 are loaded from their Hugging Face files, and add_special_tokens=False stops them adding a start-of-text marker. The first loop reads the five files. The second prints, for each language, the count on every tokenizer and its ratio to the English count on the same tokenizer.

Run on an Apple M1 Max laptop:

           o200k_base     cl100k_base           Qwen3         Llama 3
en        203 (1.00x)     204 (1.00x)     209 (1.00x)     204 (1.00x)
es        258 (1.27x)     308 (1.51x)     312 (1.49x)     308 (1.51x)
hi        310 (1.53x)     872 (4.27x)     826 (3.95x)     477 (2.34x)
ja        344 (1.69x)     432 (2.12x)     309 (1.48x)     312 (1.53x)
te        379 (1.87x)    1496 (7.33x)    1293 (6.19x)    1496 (7.33x)

On o200k_base this Telugu answer is 1.87 times the English one (379 / 203), not 2.60. Hindi is 1.53 instead of 1.67; Spanish and Japanese land close to the published ratios. Two things probably pull the ratio down. This answer is full of numbers and units, such as "220 C", "45 ml" and "step 11", which cost about the same in every language. And the lab's translations were not checked by native speakers. So the course keeps the published declaration text, which professional translators produced, for its main numbers, and Cookvale's planning sizes stay 231 and 601. The lesson of the second run is the one to keep: the ratio depends on the text. Measure your own answers in your own languages before you budget for them. The order of the languages on o200k_base holds in both runs, and so does the tokenizer story: this Telugu answer still costs 7.33 times English on cl100k_base and Llama 3.

Four Readings of the Table

The tokenizer is a pricing decision. Look down the Telugu row. The same Telugu paragraph costs 2.60 times English on o200k_base and 9.71 times on cl100k_base and Llama 3. If Cookvale ran on a model with Llama 3's tokenizer, the Telugu answer would be roughly:

231 x 9.71 = 2,243 tokens, about 2,240

Same advice, almost ten times the tokens of the English answer.

The tokenizer reflects whoever built it. Look at Chinese on Qwen3: 0.89. Chinese is cheaper than English there. Qwen is built by a lab in China, and the simplest reading is that its tokenizer's training text held far more Chinese than OpenAI's did, so Chinese strings got merged early. Llama 3's tokenizer is built on OpenAI's: the Llama 3 paper says its vocabulary "combines 100K tokens from the tiktoken tokenizer with 28K additional tokens to better support non-English languages." That is why Llama 3 and cl100k_base give the same English count and the same Telugu ratio, 9.71. The 28K extra tokens are the likely reason Hindi drops from 5.60 on cl100k_base to 2.95 on Llama 3.

Newer tokenizers fix a lot of this. OpenAI's move from cl100k_base to o200k_base cut Hindi from 5.60 times English to 1.67 times. A bigger vocabulary has room for more strings from more languages.

The gap is published, not only measured here. Petrov and co-authors measured tokenizers on the same 2,000 sentences in 200 languages (NeurIPS 2023): "The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support." Ahia and co-authors reached the same result in 2023: "speakers of a large number of the supported languages are overcharged while obtaining poorer results."

For Cookvale, the payer is Priya's company. A Telugu user's answer costs 2.6 times an English user's, and the Telugu user also waits longer, because every output token takes time to produce (lesson 4 shows how much).

A Token Is Not a Word

You might expect a token to be a word. Many people say "tokens" and mean "words".

It is wrong, and the table shows why. A token is whatever string the merge rules produced: sometimes a whole word with its leading space, sometimes part of a word, sometimes one byte of a letter. OpenAI's tokenizer page gives the usual rule for English: "A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words)." On o200k_base the research measured a little better for English: 0.87 words per token on formal prose, 0.90 on a casual paragraph. Other languages need 1.2 to 2.6 times the tokens of English for the same meaning on o200k_base, and up to 9.7 times on older tokenizers.

The right model: a token is a frequent byte string from the tokenizer's training text. Words per token is a planning rule for English prose only. It is not a law, and it does not apply to other languages, to code or to JSON.

The Colleague's Estimate

Now the second puzzle. The colleague counted 9,000 words in the system prompt and a typical recipe with its notes, then used the rule of thumb:

9,000 words / 0.75 words per token = 12,000 tokens

The real prompt was 12,060 tokens. Here is where those tokens actually came from:

system prompt + recipe, English prose
  9,000 words / 0.90 words per token     = 10,000 tokens   (6,000 + 4,000)
tool definitions, JSON plus the provider's
  tool instructions; nobody counted them
  as "words"                             =  2,000 tokens
user question                            =     60 tokens
total                                    = 12,060 tokens

Two errors cancelled out. The rule of thumb overestimated the prose by 2,000 tokens. The colleague forgot the tool definitions, which are also 2,000 tokens. When Priya adds two more tools or a Hindi system prompt, the same method will be wrong by an unknown amount. The fix is to count, shown later in this lesson.

Counts Do Not Carry Over Between Models

You might expect a token count to be a property of the text: count it once, and the number holds everywhere.

It is wrong, even inside one company. Anthropic's token counting page says: "Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape. Recount prompts against the model you plan to use rather than reusing counts measured against earlier models."

Applied to Cookvale, as an illustration: if the same prompt moved to a tokenizer that makes 30 percent more tokens,

12,060 x 1.30 = 15,678 tokens, about 15,700

The price per token would apply to the bigger number. So a model with a lower price per token can cost more per call, if its tokenizer cuts your text into more pieces.

The right model: a token count belongs to a text and a tokenizer together. Change either one and the count changes. Anthropic does not publish its tokenizer at all, so for its models the only way to count is to ask the provider (shown below).

Recap: the training text of a tokenizer decides which languages are cheap. On o200k_base, Telugu costs 2.6 times English; on older tokenizers, almost 10 times. Token counts are tied to one tokenizer.

What Else Is Inside the 12,060

Priya's own text is the system prompt, the recipe and the question. But the model reads more than her text.

The Chat Template

Lesson 1 showed that the provider turns the list of messages into one long string before tokenizing it. The format that does this is the chat template: it wraps each message in marker tokens, special tokens that say where a turn starts, who is speaking and where the turn ends. The model was trained on text in exactly this format.

Open models publish their template, so you can see it. Here is the real Qwen3 render of a two-message list (a one-line system prompt and one question), from the research for this course:

<|im_start|>system
You are a helpful support agent.<|im_end|>
<|im_start|>user
Where is my order A-1042?<|im_end|>
<|im_start|>assistant

<|im_start|> and <|im_end|> are single tokens (IDs 151644 and 151645). The words "system", "user" and "assistant" and the line breaks are tokens too. The two messages' text alone is 18 tokens. The rendered string is 31 tokens. The template added 13 tokens for two short messages, and you never wrote any of them.

Cookvale's own two messages from lesson 1 go through the same template, with thinking turned off. This script prints the token IDs, then each ID turned back into its text:

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
 
messages = [
    {"role": "system", "content": "You are Cookvale's cooking assistant. Answer in one or two short sentences."},
    {"role": "user", "content": "Can I use butter instead of oil?"},
]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
 
ids = tok(text)["input_ids"]
print(ids)
print([tok.decode([i]) for i in ids])

Run on an Apple M1 Max laptop:

[151644, 8948, 198, 2610, 525, 12514, 77729, 594, 17233, 17847, 13, 21806, 304, 825, 476, 1378, 2805, 22870, 13, 151645, 198, 151644, 872, 198, 6713, 358, 990, 14100, 4518, 315, 5590, 30, 151645, 198, 151644, 77091, 198, 151667, 271, 151668, 271]
['<|im_start|>', 'system', '\n', 'You', ' are', ' Cook', 'vale', "'s", ' cooking', ' assistant', '.', ' Answer', ' in', ' one', ' or', ' two', ' short', ' sentences', '.', '<|im_end|>', '\n', '<|im_start|>', 'user', '\n', 'Can', ' I', ' use', ' butter', ' instead', ' of', ' oil', '?', '<|im_end|>', '\n', '<|im_start|>', 'assistant', '\n', '<think>', '\n\n', '</think>', '\n\n']

The two messages' text alone is 24 tokens (lesson 1's script counted it); the rendered string is 41, the length of the list above. Thirteen of the 17 extra tokens are the same markers as above: <|im_start|>, a role word and a line break before each message, <|im_end|> and a line break after it. The other four are the empty thinking block that enable_thinking=False adds: <think> (151667), a blank line (\n\n, 271), </think> (151668) and another blank line. Notice also that "Cookvale" is two tokens, Cook and vale. A made-up name costs more than a common word.

Closed providers do the same thing on their servers. OpenAI's token counting guide says the count "includes formatting tokens used to represent request structure, such as message roles and boundaries. These tokens might not appear in the text or fields you tokenize locally."

Output has hidden tokens too. OpenAI says some models "generate tokens used to format or delimit response channels, tool calls, and other message structure", so "the reported output or completion token count can be higher than the number of visible tokens".

Tool Definitions Are Text

Predict before reading on. Take a short message and add one small tool definition: a name, a one-line description and one argument. How many tokens does it add?

To answer, look at what a tool becomes. The model has no special input for tools. The template writes each tool definition into the prompt as text, with instructions on how to call it. Here is part of the real Qwen3 render with one tool, from the research for this course:

# Tools
 
You may call one or more functions to assist with the user query.
 
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_order", "description": "Look up an order", "parameters": {"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}}}
</tools>
 
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
...

Every one of those tokens is input, on every call. Anthropic's token counting page has a clean measurement. A system prompt ("You are a scientist") and the message "Hello, Claude" count as:

{ "input_tokens": 14 }

One small get_weather tool (a name, a one-line description, one location argument) and a one-line weather question count as:

{ "input_tokens": 403 }

Anthropic's pricing page explains most of the gap: "When you use tools, the API also automatically includes a special system prompt for the model that enables tool use." Its table lists that hidden system prompt at 286 tokens on its current models (with tool_choice set to auto or none).

with one tool                                403 tokens
hidden tool-use system prompt              - 286 tokens
the tool's JSON definition + the question  = 117 tokens
 
without tools (different short message)       14 tokens

So one small tool took a short request from tens of tokens to about 400. Most of the cost was instructions you never see.

Cookvale, Band by Band

Now every token of Cookvale's turn 1 can be accounted for. The order is the order providers render: tools, then the system prompt, then the recipe, then the question.

tool definitions   2,000 tokens   8 tools (find_recipes, get_recipe, scale_recipe,
                                  convert_units, substitute_ingredient, set_timer,
                                  add_to_shopping_list, get_pantry), JSON schemas,
                                  plus the provider's tool instructions
system prompt      6,000 tokens   voice and house rules, food safety, units,
                                  six worked example exchanges
retrieved recipe   4,000 tokens   the recipe being cooked (about 900 tokens)
                                  plus three technique notes the app attaches
user question         60 tokens   average across all five languages
 
total input        2,000 + 6,000 + 4,000 + 60 = 12,060 tokens

The user typed 60 of those tokens. The other 12,000 are Cookvale's own text: the same 8,000 tokens of tools and system prompt for every user, plus the recipe. Every tool Priya adds costs tokens on every call, whether the user needs that tool or not.

Format Changes the Count

Here is the main prediction of this lesson. Take some time with it.

Cookvale's app stores each recipe as JSON. The app can paste the recipe into the prompt as pretty JSON (indented, one field per line), as minified JSON (no spaces), or as plain text. Same information. How much more does the pretty JSON cost than the plain text?

Think about what JSON is made of. Every field name appears again in every item: "quantity" is written once per ingredient. Every name sits in quotes. Every value is followed by a comma. Braces and brackets open and close each object. Pretty JSON adds runs of spaces at the start of every line, and a run of spaces is its own token. None of those characters are frequent English words, so the merge rules cut them into small pieces.

Here is a real JSON fragment as o200k_base tokens, from the research for this course:

text     {"user_id": 4521, "status": "shipped"}
 
tokens   [{"] [user] [_id] [":] [ ] [452] [1] [,] [ "] [status] [":] [ "] [sh] [ipped] ["}]
         15 tokens; [ ] is a single space, [ "] is a space and a quote

{" and ": are single tokens. The number 4521 is two. The common word "status" is one, but "shipped" is two. The structure costs about as many tokens as the data.

The research measured the same 50 order records in three formats (fields order_id, customer_email, status, total_cents, created_at):

pretty JSON / CSV           = 3,187 / 1,447 = 2.20
saving from minifying alone = 1 - 2,238 / 3,187 = 0.298, about 30 percent

Pretty JSON cost 2.2 times the same data as CSV. CSV says each field name once, in the header. JSON says it on every row. CSV is the plain-text form the research measured; the counts for a Cookvale recipe come from the lab:

The lab's script builds one Cookvale recipe (lemon herb chicken thighs with crispy potatoes: 13 ingredients and 9 steps) as a Python dictionary, then counts it three ways: json.dumps(recipe, indent=2) for pretty JSON, json.dumps(recipe, separators=(",", ":")) for minified JSON, and plain text with one line per ingredient and per step. Run on an Apple M1 Max laptop:

format         | chars | o200k tokens | vs plain | Qwen3 tokens
pretty JSON    |  1875 |          570 |    2.10x |          590
minified JSON  |  1345 |          359 |    1.32x |          377
plain text     |   940 |          272 |    1.00x |          294
 
first 24 o200k tokens of the pretty JSON: ['{\n', ' ', ' "', 'id', '":', ' "', 'rc', 'p', '_', '8', 'f', '21', 'c', '",\n', ' ', ' "', 'title', '":', ' "', 'L', 'emon', ' herb', ' chicken', ' thighs']

Pretty JSON cost 2.10 times the plain text (570 / 272). Minifying alone saved 37 percent (1 - 359 / 570). The first tokens show where the extra goes: on every line, a line break, two spaces of indent, a quote, the key, and a quote with a colon.

There is a trade-off: a model may read a flat format less reliably for deeply nested or very wide data. A recipe (a title, ingredients, steps) is neither, so plain text suits it. Tool definitions must stay JSON schemas, because providers require that format.

The Same Letters, Twice the Tokens

One more surprise, for text users type. Unicode can store many accented letters two ways. NFC stores "é" as one character; NFD stores it as "e" plus a separate accent mark. They look identical, but they are different bytes, so they become different tokens.

The research found a real case. The Vietnamese translation of the Universal Declaration of Human Rights was stored in NFD. On o200k_base it was 6,950 tokens. The same text converted to NFC was 3,093 tokens.

NFD / NFC = 6,950 / 3,093 = 2.25

Same visible text, 2.25 times the tokens. Qwen3's tokenizer converts text to NFC itself, so it counted 3,032 either way. tiktoken and Llama 3 do not convert.

The lab ran the same check on Cookvale text: the Spanish butter answer and two recipe titles, each in NFC and in NFD, counted on o200k_base and on Qwen3. Run on an Apple M1 Max laptop:

text               | form | chars | o200k tokens | Qwen3 tokens | looks identical
Spanish answer     | NFC  |   972 |          258 |          312 | True
Spanish answer     | NFD  |   991 |          284 |          312 | True
recipe title (fr)  | NFC  |    53 |           17 |           20 | True
recipe title (fr)  | NFD  |    60 |           24 |           20 | True
recipe title (vi)  | NFC  |    43 |           17 |           18 | True
recipe title (vi)  | NFD  |    56 |           30 |           18 | True
 
o200k pieces, NFC: Cr|ème| brû|lée| à| la| van|ille|,| café| et| noix| de| coco| râ|p|ée
o200k pieces, NFD: Cre|<U+0300>|me| bru|<U+0302>|le|<U+0301>|e| a<U+0300>| la| van|ille|,| cafe|<U+0301>| et| noix| de| coco| ra|<U+0302>|pe|<U+0301>|e
(<U+0300>, <U+0301>, <U+0302> are the combining accent marks that NFD stores as separate characters)

Every row looks identical on screen. On o200k_base the Spanish answer costs 10 percent more in NFD (284 / 258), and the Vietnamese title 76 percent more (30 / 17), because Vietnamese puts more accents on each word. The last two lines show why: in NFD each accent mark becomes a token of its own. The Qwen3 column does not move.

Text copied from some PDFs, from macOS file names and from some keyboards arrives as NFD. Converting user text to NFC before sending is one line of Python:

import unicodedata
 
clean = unicodedata.normalize("NFC", user_text)  # combine letter + accent into one character

Recap: the template, the tool instructions and the format of your data are all tokens. One small tool took a request from 14 to 403 tokens, and pretty JSON cost 2.2 times CSV for the same records.

What Counts Against the Context Window

The context window is the most tokens one call can hold. It is not only a limit on the prompt. It covers the prompt and the answer together. Anthropic's definition: "The "context window" refers to all the text a language model can reference when generating a response, including the response itself." And in more detail: "Everything in the request counts toward the context window: the system prompt, every message in messages (including tool results, images, and documents), and your tool definitions. The output Claude generates for the turn, including its extended thinking, counts too."

"Extended thinking" means thinking tokens: output tokens the model writes before the answer. They take space in the window and are billed as output, even when the user never sees them.

OpenAI's model pages make the sharing visible as two numbers. For its current mid-size model (checked 2026-10-18):

context window             1,050,000 tokens
maximum input tokens         922,000 tokens
room kept for the output   1,050,000 - 922,000 = 128,000 tokens

The 128,000 matches the model's maximum output. The window is one space, and input and output share it.

Every Turn Sends Everything Again

Lesson 1 showed that the API keeps nothing between calls. To continue a chat, the app sends the whole conversation again. Anthropic's docs describe a turn's input as: "Contains all previous conversation history plus the current user message".

A Cookvale chat has 4 turns on average. Each later turn adds the previous answer (300 tokens) and the new question (60 tokens):

turn 1 input = 2,000 + 6,000 + 4,000 + 60 = 12,060
turn 2 input = 12,060 + 300 + 60          = 12,420
turn 3 input = 12,420 + 360               = 12,780
turn 4 input = 12,780 + 360               = 13,140
 
input billed per chat  = 12,060 + 12,420 + 12,780 + 13,140 = 50,400 tokens
output billed per chat = 4 x 300                           =  1,200 tokens

Predict before reading on. Turn 4 sends 13,140 tokens. How many of them has the provider seen before, in this same chat?

All of them except the new question:

turn 4 input                          13,140 tokens
  new in turn 4 (the question)      -     60 tokens
  already sent or written before    = 13,080 tokens
  share already seen                = 13,080 / 13,140 = 99.5 percent

The 13,080 are turn 3's whole input plus the model's own turn 3 answer. Priya pays the input price for all 13,140, and the model reads them all again.

At the end of turn 4 the context holds 13,140 input tokens plus the 300-token answer: 13,440 tokens.

13,440 / 1,050,000 = 0.0128, about 1.3 percent of the window

Cookvale is nowhere near the context window. The limit that matters to Priya is the bill, and later lessons show it also matters to how much GPU memory each conversation needs. A request that does not fit is refused, not cut short. Anthropic: "If the input alone already exceeds the model's context window, the API returns a 400 invalid_request_error ("prompt is too long") on every model."

Counting Before You Send

The colleague's mistake had a simple cause: an estimate instead of a count. There are two ways to count.

The provider's counting endpoint. It takes the same request you would send and returns the number the provider will bill, template and tools included. Anthropic's (verbatim from its docs):

curl https://api.anthropic.com/v1/messages/count_tokens \
 -H "x-api-key: $ANTHROPIC_API_KEY" \
 -H "content-type: application/json" \
 -H "anthropic-version: 2023-06-01" \
 -d '{
 "model": "claude-opus-5-5",
 "system": "You are a scientist",
 "messages": [{
 "role": "user",
 "content": "Hello, Claude"
 }]
 }'

{ "input_tokens": 14 }

The model field matters: the endpoint counts with that model's tokenizer. Anthropic says counting is "free to use" and that "The token count is an estimate." OpenAI has the same kind of endpoint at POST /v1/responses/input_tokens. Both need an API key; this course does not require one.

A local tokenizer. tiktoken and the Hugging Face tokenizers library run on your laptop with no key. They are exact for plain text in the encoding you load, and they miss what the provider adds. OpenAI's guide: "Local tokenizers like tiktoken work for plain text, but they have limitations", including "Tools and schemas add tokens that are hard to count locally". For Anthropic's models there is no local tokenizer at all.

So use a local tokenizer to compare texts (English against Telugu, JSON against plain text) and the provider's endpoint to know the real number for a request.

Recap: input and output share the context window, thinking included. Every turn re-sends the whole chat, so Cookvale's turn 4 is 13,140 tokens, 99.5 percent of them already seen. Count with the provider's endpoint, compare with a local tokenizer.

What a Token Costs

Providers price tokens per million. Here is the price card for the mid-size model Cookvale uses:

mid-size model, per million tokens (checked 2026-10-18)
  input     $2.00
  output   $10.00   (5 times input)

Claude Sonnet 5.5 and GPT-6.1 Sol both list exactly these prices on 2026-10-18. Later lessons add the lines for cached input and batch.

One Call

Turn 1 of a chat:

input    12,060 tokens x $2  / 1,000,000 = $0.02412
output      300 tokens x $10 / 1,000,000 = $0.00300
total                                      $0.02712, about 2.7 cents
 
input cost / output cost = 0.02412 / 0.00300 = 8.04, about 8 times

Each output token costs five times an input token. But there are 40 times more input tokens (12,060 against 300), so the input line is 8 times bigger.

One Day

50,000 chats a day, each with 50,400 input tokens and 1,200 output tokens:

input per day    50,000 chats x 50,400 = 2,520,000,000 tokens
output per day   50,000 chats x  1,200 =    60,000,000 tokens
 
input    2,520,000,000 x $2  / 1,000,000 = $5,040
output      60,000,000 x $10 / 1,000,000 =   $600
total                                      $5,640 a day
 
input share       5,040 / 5,640 = 0.894, about 89 percent
per month         $5,640 x 30 = $169,200

Most of Cookvale's bill is tokens the provider has already seen: the same 8,000-token prefix for every user and the same chat re-sent on every turn.

What the Languages Add

The 300-token average answer is not a design choice. It is the language mix, weighted by each language's share of chats:

English    0.50 x 231 = 115.50
Spanish    0.20 x 282 =  56.40
Hindi      0.15 x 386 =  57.90
Japanese   0.10 x 407 =  40.70
Telugu     0.05 x 601 =  30.05
average answer        = 300.55, about 300 tokens

If every user wrote in English, the average answer would be 231 tokens. Compare the two:

output per day, real mix     200,000 calls x 300 = 60,000,000 tokens   $600
output per day, all English  200,000 calls x 231 = 46,200,000 tokens   $462
added by the language mix                                              $138 a day
 
one English answer   231 x $10 / 1,000,000 = $0.00231
one Telugu answer    601 x $10 / 1,000,000 = $0.00601

The language mix adds 30 percent to Cookvale's output tokens (300 / 231 = 1.30). It is a real line in the budget. The users chose their languages; the tokenizer decided what each language costs.

So the rule for this lesson: you pay for tokens, and the language, the format and the tokenizer decide how many tokens a text becomes.

Check today. On 2026-10-18, OpenAI's tiktoken mapped its GPT-5 models to o200k_base and did not yet list the GPT-6 models; Anthropic's Claude 4.7 and later models used a newer tokenizer that makes about 30 percent more tokens than earlier Claude models, and Anthropic publishes no local tokenizer. Tokenizers change with model generations, so the language ratios in this lesson move too. Check the current model-to-encoding map: https://github.com/openai/tiktoken/blob/main/tiktoken/model.py

The Takeaway

1. A token is a frequent byte string from a tokenizer's training text. Byte pair encoding builds the tokenizer by merging the most frequent pair of symbols, again and again. Frequent strings become one token; rare strings stay in pieces.
2. So a tokenizer is cheap for the languages that filled its training text. The same Cookvale answer is 231 tokens in English and 601 in Telugu on o200k_base (2.6 times), and would be about 2,240 on Llama 3's tokenizer (9.7 times).
3. A token is not a word. Words per token is a rough rule for English prose only. Token counts belong to one tokenizer: Anthropic's newer tokenizer makes about 30 percent more tokens for the same text.
4. The model reads more than your text: template markers (13 extra tokens for two short Qwen3 messages), tool instructions (one small tool took a request from 14 to 403 tokens) and the format of your data (pretty JSON cost 2.2 times CSV).
5. Input and output share the context window, thinking tokens included. Every turn re-sends the whole chat: 12,060, 12,420, 12,780, 13,140.
6. Count with the provider's counting endpoint for the real number; use a local tokenizer to compare texts.
7. Cookvale's turn 1 costs $0.02712, with input 8 times the output cost. A day costs $5,640, 89 percent of it input. The language mix adds 30 percent to the output tokens.

The one sentence to keep: you pay for tokens, not words, and the number of tokens a text becomes depends on its language, its format and the model's tokenizer.

An open question for later: every turn re-sends twelve thousand tokens the provider has already read. Does it have to read them all again? The KV cache shows what the model saves inside one call, and Paged memory and prompt caching shows what a provider can keep between calls.

References

Papers:

• Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016 (arXiv 2015): byte pair encoding for language models
• Radford et al., Language Models are Unsupervised Multitask Learners, OpenAI 2019, section 2.2: byte-level BPE, "a base vocabulary of size 256", the pre-split rule
• Llama Team, The Llama 3 Herd of Models, 2024: the 128K vocabulary built on tiktoken's 100K tokens
• Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023
• Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, EMNLP 2023

Provider documentation, checked 2026-10-18:

• Token counting (Claude API docs): the count_tokens endpoint, the 14 and 403 examples, "estimate", "free to use", the newer tokenizer and the 30 percent note
• Pricing (Claude API docs): the tool use system prompt (286 tokens on current models), the tokenizer note, the price card
• Context windows (Claude API docs): the definition "including the response itself", what counts toward the window, "prompt is too long"
• Messages API reference (Claude API docs): the usage note on counts that do not match visible content
• Counting tokens (OpenAI API docs): POST /v1/responses/input_tokens, formatting tokens, the limits of local tokenizers, hidden output formatting tokens
• GPT-6.1 Sol model page (OpenAI API docs): 1,050,000-token context window, 922,000 maximum input tokens
• OpenAI Pricing: the price card
• OpenAI Tokenizer: the "~4 characters" rule of thumb

Tokenizers and tools:

• tiktoken 0.14.0 and its model-to-encoding map ("gpt-5": "o200k_base"), checked 2026-10-18
• Hugging Face tokenizers 0.23.3; Qwen3 tokenizer.json and chat template in tokenizer_config.json; Llama 3.1 tokenizer and template from the ungated mirror unsloth/Meta-Llama-3.1-8B-Instruct
• Tiktokenizer: colours tokens and IDs in the browser
• Unicode UDHR project: the parallel texts behind the language table, NFC-normalized for the table

Measurements: the language table, the strawberry and digit splits, the JSON fragment, the 50-record format table, the Qwen3 and Llama 3.1 template renders and the Vietnamese NFC and NFD counts were made by the research for this course on an Apple M1 Max, 2026-10-18, with tiktoken 0.14.0, tokenizers 0.23.3 and transformers 5.20.1. The Cookvale answer table, the Cookvale template render, the recipe JSON counts and the Cookvale NFC and NFD counts come from the course's lab on the same laptop (tiktoken 0.14.0, transformers 5.20.1; the Llama 3 tokenizer from the ungated unsloth/Llama-3.2-1B-Instruct files); the lab's translations were not reviewed by native speakers. The toy BPE run and every other Cookvale figure are worked in this lesson.