Tokens: what you pay for

Cookvale is a recipe app. Its cooking assistant answers questions while a person cooks: can I swap this, how long does this take, is this done. Priya is the engineer who owns it. Last week Cookvale launched the assistant in five languages, and this morning Priya is looking at her usage dashboard.

Two rows on it make no sense to her. A user cooking in English asked whether the rice was done, and got an answer of 231 output tokens. A user cooking in Telugu asked the same question about the same recipe, and got the same advice, in Telugu. That answer was 601 output tokens. Same meaning, same model, same price per token. The Telugu answer cost 2.6 times as much.

Telugu answer / English answer = 601 / 231 = 2.60

There is a second puzzle on her desk. Before launch, a colleague estimated the size of the prompt: "our system prompt and a typical recipe come to about 9,000 words, so the prompt is about 12,000 tokens." The first real usage log said 12,060 input tokens. The estimate was right. But, as you will see in this lesson, it was right for the wrong reason, and the next estimate made the same way will be wrong.

Both puzzles have the same root. Priya pays for tokens, not words. So what is a token, and who decides how many tokens a piece of text becomes?

By the end of this lesson you will be able to explain how a tokenizer is built, predict which texts cost more, and say why a token count only holds for one tokenizer.

What a Token Is

A token is one piece of text that the model reads or writes. The model never sees letters or words. It sees a list of integers, one per token. Each integer is a token ID. The program that cuts text into tokens, and turns tokens back into text, is the tokenizer. The full list of tokens a tokenizer knows is its vocabulary. Each model family has its own tokenizer: o200k_base has about 200,000 tokens in its vocabulary, Llama 3 has 128,000, and Qwen3 has 151,643 regular tokens.

What happens in one model call showed that a model call starts by turning the prompt into token IDs and ends by turning the output IDs back into text. This lesson opens that step.

How a Tokenizer Is Built

Almost every current tokenizer is built with byte pair encoding (BPE). It is a short procedure:

1. Start with a very large collection of training text.
2. Write every piece of text as single symbols.
3. Count every pair of neighbouring symbols across the whole collection.
4. Take the most frequent pair and make it one new symbol. Write that rule down.
5. Repeat from step 3, tens of thousands of times.

The list of rules, in order, is the tokenizer. To tokenize new text later, the program applies the same rules in the same order.

Here is the procedure on a tiny collection: the word "stir" three times, "stew" two times and "salt" once. Words are counted separately, the way real tokenizers first split text at spaces.

start      s t i r (x3)    s t e w (x2)    s a l t (x1)
 
round 1    pair counts: s t = 3 + 2 = 5, t i = 3, i r = 3, t e = 2, e w = 2,
           s a = 1, a l = 1, l t = 1
           most frequent: s t (5)            new symbol "st"
round 2    st i = 3, i r = 3 (a tie; the first one found wins)
                                             new symbol "sti"
round 3    sti r = 3                         new symbol "stir"
round 4    st e = 2, e w = 2 (a tie again)   new symbol "ste"
round 5    ste w = 2                         new symbol "stew"
 
after 5 rounds
  "stir"  -> 1 token    (frequent)
  "stew"  -> 1 token    (frequent)
  "salt"  -> 4 tokens   s | a | l | t   (seen once, never merged)

That last line is the whole idea. Strings that were frequent in the training text become one token. Rare strings stay in pieces. A real tokenizer runs the same loop with a much bigger collection and many more rounds. The Llama 3 tokenizer file holds 280,147 merge rules.

Two details make real tokenizers work on any text.

First, the starting symbols are bytes, not letters. Every text on a computer is stored as bytes, usually in the UTF-8 format, so a byte-level tokenizer can read any language, any emoji and any typo. The GPT-2 paper put it this way: "a byte-level version of BPE only requires a base vocabulary of size 256." There is never an unknown character. Notice what this means for different languages. In UTF-8, an English letter is 1 byte. A Spanish "ñ" is 2 bytes. A Hindi letter, a Telugu letter and a Japanese character are each 3 bytes. Before a single merge, a Telugu word already starts as about three times more symbols than an English word with the same number of letters.

Second, before merging, the tokenizer splits text into chunks with a fixed pattern (a regular expression) and never merges across those chunks. The GPT-2 authors explain why: without it, BPE learned "many versions of common words like dog since they occur in many variations such as dog. dog! dog?", so they "prevent BPE from merging across character categories". The pattern keeps a leading space attached to the word that follows it. On o200k_base, " stir" with a space is one token, and "stir" without one is two: st ir.

Strawberry, With and Without a Space

That last detail explains a failure many people have seen: models are bad at counting the letters in a word. Run this on any laptop:

import tiktoken  # tiktoken 0.14.0
 
enc = tiktoken.get_encoding("o200k_base")
 
for text in ["strawberry", " strawberry"]:
    ids = enc.encode(text)                      # text -> list of token IDs
    pieces = [enc.decode([i]) for i in ids]     # each ID back to its text
    print(repr(text), len(ids), pieces)

get_encoding loads the o200k_base rules. encode applies them and returns token IDs. The next line turns each ID back into its text, so you can see the cuts. The research for this course measured these splits with tiktoken 0.14.0 (2026-10-18):

In the middle of a sentence, "strawberry" follows a space, so the model receives one integer, with no letters to count. At the start of a line it receives three different integers. Neither input shows the letter "r". That is why spelling, counting letters and reversing strings are weak spots: the model works on IDs, not characters.

Numbers have their own rules. o200k_base and Llama 3 cut digits into groups of up to three: "1234567" becomes 123 456 7. Qwen3 cuts every digit alone, so the same number is 7 tokens. A recipe full of grams and temperatures costs a little more on Qwen3 for that reason.

You can see all of this live in a browser: Tiktokenizer colours each token of any text you paste and shows its ID.

Recap: a tokenizer is a list of merge rules learned from training text. Frequent strings become one token, rare strings stay in pieces, and the pieces are what you pay for.

Why Telugu Costs 2.6 Times English

Before reading on, predict. A Cookvale answer is translated into Telugu with exactly the same meaning. Will it be more tokens, fewer, or the same?

The procedure you just saw gives the answer. Merge rules come from the training text. The more often a string appeared in that text, the earlier it got merged into one token. So a tokenizer is cheap for the languages that filled its training text and expensive for the rest. English words like " the", " recipe" and " minutes" became single tokens early. Telugu words appeared far less often, so most of them were never merged, and each one stays as several pieces. Each piece starts from 3-byte letters.

The Measured Table

The research for this course counted the same text in many languages: the Universal Declaration of Human Rights, which exists in parallel translations. The ratio is the token count for a language divided by the token count for English, with the same tokenizer. Measured on 2026-10-18 on an Apple M1 Max with tiktoken 0.14.0 and the Qwen3 and Llama 3 tokenizer.json files:

Cookvale's answer sizes come from the o200k_base column. Start from the English answer, 231 tokens, and multiply by each ratio:

English    231 x 1.00 = 231 tokens
Spanish    231 x 1.22 = 281.8, about 282 tokens
Hindi      231 x 1.67 = 385.8, about 386 tokens
Japanese   231 x 1.76 = 406.6, about 407 tokens
Telugu     231 x 2.60 = 600.6, about 601 tokens

That is Priya's dashboard: 231 and 601.

A published parallel text is the fair way to compare languages, but your own text can land somewhere else. The lab for this course counted one real Cookvale answer, the reply to "Can I use butter instead of oil?" (164 English words), in all five languages on all four tokenizers. Each file holds one translation of the same answer:

import tiktoken                          # tiktoken 0.14.0
from transformers import AutoTokenizer   # transformers 5.20.1
 
o200k = tiktoken.get_encoding("o200k_base")
cl100k = tiktoken.get_encoding("cl100k_base")
qwen = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
llama = AutoTokenizer.from_pretrained("unsloth/Llama-3.2-1B-Instruct")  # the Llama 3 tokenizer
 
count = {
    "o200k_base": lambda t: len(o200k.encode(t)),
    "cl100k_base": lambda t: len(cl100k.encode(t)),
    "Qwen3": lambda t: len(qwen.encode(t, add_special_tokens=False)),
    "Llama 3": lambda t: len(llama.encode(t, add_special_tokens=False)),
}
 
langs = ["en", "es", "hi", "ja", "te"]
texts = {}
for lang in langs:
    with open(f"answer_{lang}.txt", encoding="utf-8") as f:
        texts[lang] = f.read().strip()
 
print("     " + "".join(f"{name:>16}" for name in count))
for lang in langs:
    cells = ""
    for name, n in count.items():
        tokens = n(texts[lang])
        cells += f"{tokens:>8} ({tokens / n(texts['en']):.2f}x)"
    print(f"{lang:5}" + cells)

count maps each tokenizer's name to a small function that returns a token count; Qwen3 and Llama 3 are loaded from their Hugging Face files, and add_special_tokens=False stops them adding a start-of-text marker. The first loop reads the five files. The second prints, for each language, the count on every tokenizer and its ratio to the English count on the same tokenizer.

Run on an Apple M1 Max laptop:

           o200k_base     cl100k_base           Qwen3         Llama 3
en        203 (1.00x)     204 (1.00x)     209 (1.00x)     204 (1.00x)
es        258 (1.27x)     308 (1.51x)     312 (1.49x)     308 (1.51x)
hi        310 (1.53x)     872 (4.27x)     826 (3.95x)     477 (2.34x)
ja        344 (1.69x)     432 (2.12x)     309 (1.48x)     312 (1.53x)
te        379 (1.87x)    1496 (7.33x)    1293 (6.19x)    1496 (7.33x)

On o200k_base this Telugu answer is 1.87 times the English one (379 / 203), not 2.60. Hindi is 1.53 instead of 1.67; Spanish and Japanese land close to the published ratios. Two things probably pull the ratio down. This answer is full of numbers and units, such as "220 C", "45 ml" and "step 11", which cost about the same in every language. And the lab's translations were not checked by native speakers. So the course keeps the published declaration text, which professional translators produced, for its main numbers, and Cookvale's planning sizes stay 231 and 601. The lesson of the second run is the one to keep: the ratio depends on the text. Measure your own answers in your own languages before you budget for them. The order of the languages on o200k_base holds in both runs, and so does the tokenizer story: this Telugu answer still costs 7.33 times English on cl100k_base and Llama 3.

Four Readings of the Table

The tokenizer is a pricing decision. Look down the Telugu row. The same Telugu paragraph costs 2.60 times English on o200k_base and 9.71 times on cl100k_base and Llama 3. If Cookvale ran on a model with Llama 3's tokenizer, the Telugu answer would be roughly:

231 x 9.71 = 2,243 tokens, about 2,240

Same advice, almost ten times the tokens of the English answer.

The tokenizer reflects whoever built it. Look at Chinese on Qwen3: 0.89. Chinese is cheaper than English there. Qwen is built by a lab in China, and the simplest reading is that its tokenizer's training text held far more Chinese than OpenAI's did, so Chinese strings got merged early. Llama 3's tokenizer is built on OpenAI's: the Llama 3 paper says its vocabulary "combines 100K tokens from the tiktoken tokenizer with 28K additional tokens to better support non-English languages." That is why Llama 3 and cl100k_base give the same English count and the same Telugu ratio, 9.71. The 28K extra tokens are the likely reason Hindi drops from 5.60 on cl100k_base to 2.95 on Llama 3.

Newer tokenizers fix a lot of this. OpenAI's move from cl100k_base to o200k_base cut Hindi from 5.60 times English to 1.67 times. A bigger vocabulary has room for more strings from more languages.

The gap is published, not only measured here. Petrov and co-authors measured tokenizers on the same 2,000 sentences in 200 languages (NeurIPS 2023): "The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support." Ahia and co-authors reached the same result in 2023: "speakers of a large number of the supported languages are overcharged while obtaining poorer results."

For Cookvale, the payer is Priya's company. A Telugu user's answer costs 2.6 times an English user's, and the Telugu user also waits longer, because every output token takes time to produce (prefill and decode shows how much).

A Token Is Not a Word

You might expect a token to be a word. Many people say "tokens" and mean "words".

It is wrong, and the table shows why. A token is whatever string the merge rules produced: sometimes a whole word with its leading space, sometimes part of a word, sometimes one byte of a letter. OpenAI's tokenizer page gives the usual rule for English: "A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words)." On o200k_base the research measured a little better for English: 0.87 words per token on formal prose, 0.90 on a casual paragraph. Other languages need 1.2 to 2.6 times the tokens of English for the same meaning on o200k_base, and up to 9.7 times on older tokenizers.

The right model: a token is a frequent byte string from the tokenizer's training text. Words per token is a planning rule for English prose only. It is not a law, and it does not apply to other languages, to code or to JSON.

The Colleague's Estimate

Now the second puzzle. The colleague counted 9,000 words in the system prompt and a typical recipe with its notes, then used the rule of thumb:

9,000 words / 0.75 words per token = 12,000 tokens

The real prompt was 12,060 tokens. Here is where those tokens actually came from:

system prompt + recipe, English prose
  9,000 words / 0.90 words per token     = 10,000 tokens   (6,000 + 4,000)
tool definitions, JSON plus the provider's
  tool instructions; nobody counted them
  as "words"                             =  2,000 tokens
user question                            =     60 tokens
total                                    = 12,060 tokens

Two errors cancelled out. The rule of thumb overestimated the prose by 2,000 tokens. The colleague forgot the tool definitions, which are also 2,000 tokens. When Priya adds two more tools or a Hindi system prompt, the same method will be wrong by an unknown amount. The fix is to count, which counting every token shows.

Counts Do Not Carry Over Between Models

You might expect a token count to be a property of the text: count it once, and the number holds everywhere.

It is wrong, even inside one company. Anthropic's token counting page says: "Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape. Recount prompts against the model you plan to use rather than reusing counts measured against earlier models."

Applied to Cookvale, as an illustration: if the same prompt moved to a tokenizer that makes 30 percent more tokens,

12,060 x 1.30 = 15,678 tokens, about 15,700

The price per token would apply to the bigger number. So a model with a lower price per token can cost more per call, if its tokenizer cuts your text into more pieces.

The right model: a token count belongs to a text and a tokenizer together. Change either one and the count changes. Anthropic does not publish its tokenizer at all, so for its models the only way to count is to ask the provider (shown in counting every token).

Recap: the training text of a tokenizer decides which languages are cheap. On o200k_base, Telugu costs 2.6 times English; on older tokenizers, almost 10 times. Token counts are tied to one tokenizer.

The Takeaway

1. A token is a frequent byte string from a tokenizer's training text. Byte pair encoding builds the tokenizer by merging the most frequent pair of symbols, again and again. Frequent strings become one token; rare strings stay in pieces.
2. So a tokenizer is cheap for the languages that filled its training text. The same Cookvale answer is 231 tokens in English and 601 in Telugu on o200k_base (2.6 times), and would be about 2,240 on Llama 3's tokenizer (9.7 times).
3. A token is not a word. Words per token is a rough rule for English prose only. Token counts belong to one tokenizer: Anthropic's newer tokenizer makes about 30 percent more tokens for the same text.

The one sentence to keep: a tokenizer's merge rules come from its training text, and they decide how many tokens your text becomes.

An open question to carry forward: the user typed about 60 of Cookvale's 12,060 input tokens. What else is inside that prompt, and which parts did nobody write? That is the next step, counting every token.

References

Papers:

• Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016 (arXiv 2015): byte pair encoding for language models
• Radford et al., Language Models are Unsupervised Multitask Learners, OpenAI 2019, section 2.2: byte-level BPE, "a base vocabulary of size 256", the pre-split rule
• Llama Team, The Llama 3 Herd of Models, 2024: the 128K vocabulary built on tiktoken's 100K tokens
• Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023
• Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, EMNLP 2023

Provider documentation, checked 2026-10-18:

• Token counting (Claude API docs): the count_tokens endpoint, the 14 and 403 examples, "estimate", "free to use", the newer tokenizer and the 30 percent note
• OpenAI Pricing: the price card
• OpenAI Tokenizer: the "~4 characters" rule of thumb

Tokenizers and tools:

• tiktoken 0.14.0 and its model-to-encoding map ("gpt-5": "o200k_base"), checked 2026-10-18
• Hugging Face tokenizers 0.23.3; Qwen3 tokenizer.json and chat template in tokenizer_config.json; Llama 3.1 tokenizer and template from the ungated mirror unsloth/Meta-Llama-3.1-8B-Instruct
• Tiktokenizer: colours tokens and IDs in the browser
• Unicode UDHR project: the parallel texts behind the language table, NFC-normalized for the table

Measurements: the language table and the strawberry and digit splits were made by the research for this course on an Apple M1 Max, 2026-10-18, with tiktoken 0.14.0, tokenizers 0.23.3 and transformers 5.20.1. The Cookvale answer table comes from the course's lab on the same laptop (tiktoken 0.14.0, transformers 5.20.1; the Llama 3 tokenizer from the ungated unsloth/Llama-3.2-1B-Instruct files); the lab's translations were not reviewed by native speakers. The toy BPE run and every other Cookvale figure are worked in this lesson.