Where the next token is chosen

Priya runs a small open model on her laptop, Qwen3-1.7B in a compressed version of about 1 GB, and asks it the same question twice: "Name one herb to add to tomato soup." The first time it answers "Thyme", and the second time "Dill". Same file, same prompt, same laptop. Both runs made the same forward pass: a lookup, every layer, and one score for every token in the vocabulary. So where, exactly, does the choice between "Thyme" and "Dill" happen?

Where the Choice Happens

The pass ends with scores. Something must still turn 151,936 scores into one token. That something is the sampler: ordinary code that runs after the model.

From Scores to Probabilities

First the scores become probabilities, with a function called softmax. Every probability ends up between 0 and 1, they add up to 1, and a higher score always gets a higher probability. You do not need its formula. You need one property of it: the ratio between two probabilities depends only on the gap between their two scores. Each point of gap multiplies the ratio by about 2.7.

Here is a real table. It was measured for this course on the laptop model, Qwen3-0.6B in bf16 on the Apple M1 Max, with a prompt of the same shape as the herb prompt: "Name one fruit. Answer with one word.", through the chat template with thinking off.

In the 1.0 column, "Apple" is most likely, at 0.574. "Mango" and "Banana" are not single tokens in this vocabulary; they compete through their first pieces, M and Ban. Lowercase apple is a different token, with its own score.

Check the gap rule on the top two rows:

score gap, Apple vs M      22.38 - 20.88 = 1.50
measured ratio at 1.0      0.574 / 0.128 = 4.5

A gap of 1.5 points makes Apple 4.5 times as likely as M.

Temperature

Temperature is a number the sampler divides every score by, before softmax. Before you look at the other columns, predict. Raise the temperature from 1.0 to 1.5. Does the top token become more likely or less?

Work it out with the gap rule:

temperature 0.5   scores / 0.5   44.76 and 41.76   gap 3.00
                  ratio about 2.7 x 2.7 x 2.7 = 19.7
                  measured: 0.914 / 0.046 = 19.9
 
temperature 1.0   scores / 1.0   22.38 and 20.88   gap 1.50
                  measured: 0.574 / 0.128 = 4.5
 
temperature 1.5   scores / 1.5   14.92 and 13.92   gap 1.00
                  ratio about 2.7
                  measured: 0.289 / 0.106 = 2.7

Less likely. Dividing by a small number stretches every gap, so the top token takes almost everything: 0.914 at temperature 0.5. Dividing by a big number shrinks every gap, so probability spreads to tokens further down the list: Apple falls to 0.289 at 1.5.

Now the same measurement for the herb prompt, with one surprise first. On Qwen3-0.6B, the top next token for this prompt is tom, the start of "tomato", at 0.547, and five of ten draws at temperature 1.0 answer "tomato". Tomato is not a herb. A model computes scores for what tends to come next; it does not check facts. So the herb runs in this lesson use its bigger sibling, Qwen3-1.7B, in a version that stores each weight in 4 bits instead of 16, through MLX, Apple's machine learning library for its own chips (so this script needs an Apple silicon Mac). It prints the top 6 next tokens for Priya's question at three temperatures:

import mlx.core as mx                      # mlx 0.32.4, Apple silicon only
from mlx_lm import load, generate          # mlx-lm 0.32.2
from mlx_lm.sample_utils import make_sampler
 
model, tok = load("mlx-community/Qwen3-1.7B-4bit")
 
messages = [{"role": "user",
             "content": "Name one herb to add to tomato soup. Answer with one word."}]
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)
ids = tok.encode(prompt, add_special_tokens=False)
 
scores = model(mx.array(ids)[None])[0, -1].astype(mx.float32)
 
print("scores in one pass:", scores.shape[0])
top = mx.argsort(-scores)[:6].tolist()
for t in [0.5, 1.0, 1.5]:
    probs = mx.softmax(scores / t, axis=-1)
    print(f"T={t}: " + "  ".join(f"{tok.decode([i])!r} {probs[i].item():.3f}" for i in top))

load downloads the model and its tokenizer. apply_chat_template wraps the question in Qwen3's chat template, as in the first lesson; enable_thinking=False asks for a direct answer with no thinking tokens (output tokens the model writes before the answer). tok.encode turns the string into token IDs. model(...) runs one forward pass over the whole prompt; [0, -1] takes the scores at the last position, the only one that decides the next token. mx.argsort(-scores)[:6] finds the six highest. For each temperature, scores / t divides every score and mx.softmax turns the result into probabilities.

Run on an Apple M1 Max laptop:

scores in one pass: 151936
T=0.5: 'C' 0.227  'Ch' 0.227  'Bay' 0.177  'O' 0.137  'Th' 0.137  'Rose' 0.031
T=1.0: 'C' 0.139  'Ch' 0.139  'Bay' 0.122  'O' 0.108  'Th' 0.108  'Rose' 0.051
T=1.5: 'C' 0.076  'Ch' 0.076  'Bay' 0.070  'O' 0.064  'Th' 0.064  'Rose' 0.039

Read it like the fruit table: the top row, the gap to the second, and how each temperature moves probability between them. No herb dominates the way Apple did. The top token has 0.139 at temperature 1.0, against Apple's 0.574, and C and Ch have the same probability to three decimal places. The tokens are first pieces again: Th can become "Thyme", O "Oregano", Rose "Rosemary", and only the next passes decide. At 0.5 the top five tokens hold 0.905 of the probability; at 1.5 all six together hold only 0.389.

You might expect a lower temperature to make the model smarter, and a higher one more creative. It is wrong. Between the columns of the fruit table, only the sampler's division changed. The scores, the weights and the pass were identical. OpenAI's API reference describes the knob correctly: "Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic." Random or focused, not smarter or dumber. A low temperature can even hurt. Qwen3's model card warns, for its thinking mode: "DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions." The right model: temperature reshapes the probabilities the model already produced; it never changes the model.

The Draw

Now the sampler draws one token at random, weighted by the probabilities. At temperature 1.0, in 100 draws Apple comes out about 57 times, M about 13, apple about 10. If M is drawn, the next pass continues it, most likely into "Mango". These lines continue the same script and ask the herb question ten times at temperature 1.0:

sampler = make_sampler(temp=1.0)           # plain temperature 1.0: no top-p, no top-k
for seed in range(10):
    mx.random.seed(seed)
    print(generate(model, tok, prompt, max_tokens=5, sampler=sampler))

make_sampler(temp=1.0) builds a sampler that draws from the plain temperature 1.0 probabilities, with no top-p or top-k cut (below). mx.random.seed(seed) fixes the random draw, so each answer can be repeated. generate runs the loop from the first lesson, a pass, a draw, append and again, for at most 5 tokens, enough for a one-word herb.

Cumin
Cumin.
Thyme.
Thyme.
Bay leaves
Garlic.
Dill.
Clove
Thyme.
Thyme.

Same scores on every run. Different draws. "Thyme" comes out four times. C became "Cumin" twice and "Clove" once, never "Chives". "Garlic" and "Dill" start with pieces outside the top six (Gar and D): at temperature 1.0 the top six hold 0.667 of the probability, so about one draw in three lands further down. And cumin, garlic and clove are spices, not herbs. Again, the model scores what tends to follow; it does not check what is true.

Cutting the Tail: Top-p and Top-k

Below the six rows shown sit about 151,930 more tokens, each with a tiny probability. Together they are not tiny. At temperature 1.5 the top six fruit tokens add up to only about 0.62, so more than a third of the draws land further down the list, where the text can stop making sense.

Holtzman and others named this in 2019. They showed that always taking the likeliest token "leads to text that is bland and strangely repetitive", and proposed instead "sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution". That method is top-p, also called nucleus sampling: keep the smallest set of top tokens whose probabilities add up to at least p, drop the rest, and scale the kept ones so they add up to 1 again.

Top-p 0.8 on the fruit table at temperature 1.0:

add the top tokens until the total reaches 0.8
  Apple                          0.574                total 0.574
  M                              0.128                total 0.702
  apple                          0.100                total 0.802   reached 0.8, stop
 
keep 3 tokens out of 151,936; drop everything else
scale the kept ones to add up to 1 (divide by 0.802)
  Apple   0.574 / 0.802 = 0.716
  M       0.128 / 0.802 = 0.160
  apple   0.100 / 0.802 = 0.125

Now the draw can give only one of three tokens. Top-p adapts: when one token dominates it keeps very few, and when probabilities are spread out it keeps many.

Top-k keeps a fixed number, the k most likely tokens. Min-p keeps every token with at least a set fraction of the top token's probability: min-p 0.1 here keeps tokens at or above 0.0574, the same three.

Most engines run these steps in this order (details differ by engine):

scores from the pass
  -> adjustments (biases, penalties, a grammar mask)
  -> divide by temperature
  -> softmax: probabilities
  -> cut the tail (top-k, top-p, min-p), scale back to 1
  -> draw one token

A model also ships with default settings. Qwen3-0.6B has a second small file, generation_config.json. These are its sampling fields:

{
  "do_sample": true,
  "temperature": 0.6,
  "top_k": 20,
  "top_p": 0.95
}

An app that never sets a temperature gets this file's settings: sample at random, temperature 0.6, keep at most the top 20 tokens, then the top-p 0.95 nucleus.

Temperature Zero

Here is the question to stop and think about. Set the temperature to zero. Will the answer be the same every time?

...

Dividing by zero is not defined, so engines treat temperature 0 as greedy: skip the draw and take the highest score. On Priya's laptop, with one prompt running alone, the answer is yes. The research for this course ran Qwen3-0.6B twice on the same prompt on the M1 Max and measured a largest score difference of exactly 0.0. Same scores, same top token, same answer.

On a provider's server, the answer is "not always". Anthropic's API reference says so directly: "Note that even with temperature of 0.0, the results will not be fully deterministic." The reason is not in the sampler but in how a server shares a GPU between many people's calls; service tiers and temperature zero shows how.

Logprobs

Some APIs return the probability of each token they wrote as a logprob: the log of the probability. For Apple at 0.574 the logprob is -0.555; near 0 means likely, very negative means unlikely. OpenAI's API returns them when you set logprobs (and up to 20 alternatives per position with top_logprobs). Anthropic's Messages API has no logprobs parameter (checked 2026-10-18). Engineers use them as confidence scores, for example in classification.

Check today. On 2026-10-18, Anthropic's API reference said models released after Claude Opus 4.6 reject any temperature other than 1.0 and any top_k, and accept only top_p values of 0.99 or more. OpenAI's reference marked seed as "Deprecated." Closed APIs are removing sampling knobs as new model generations ship, so what you may set depends on the model. Check the parameter list: https://platform.claude.com/docs/en/api/messages/create

The model ends at scores; the sampler turns them into probabilities, reshapes them with temperature, cuts the tail with top-p or top-k, and draws, so the choice between two answers happens after the model, not inside it.

The Puzzle, Answered

The choice happened after the model, in the sampler. Priya's runs drew at temperature 1.0 with nothing cut. Th, the start of "Thyme", has a probability of 0.108. "Dill" starts with D, which is not even in the top six, yet one draw landed on it. Same scores, two different draws. With no temperature set, Qwen3's default settings in generation_config.json still sample at random (temperature 0.6, top-k 20, top-p 0.95), so the answer would still change from run to run, only among fewer candidates. At temperature zero, her laptop would give the same herb every time.

The Takeaway

1. The sampler turns scores into probabilities. Temperature divides the scores first (low stretches the gaps, high shrinks them) and never changes the model.
2. Top-p and top-k cut the tail. Then one token is drawn. Temperature zero takes the top score.
3. Alone on a laptop, temperature zero gives the same answer every time. On a provider's busy server, it does not always.

The one sentence to keep: the model only scores the next token, and the sampler draws it, so temperature and top-p change the draw, never the model.

An open question to carry forward: Llama 3.1 8B has 16 GB of weights, and every pass uses all of them. So how much arithmetic does one pass take, for one token? That is the next step, counting the work, forcing the shape.

References

Papers:

• Holtzman et al., "The Curious Case of Neural Text Degeneration", ICLR 2020: nucleus (top-p) sampling
• Nguyen et al., "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs", 2024: min-p

Model files on Hugging Face (checked 2026-10-18):

• Qwen/Qwen3-0.6B, revision c1899de289a04d12100db370d81485cdf75e47ca: config.json, generation_config.json, model.safetensors (1,503,300,328 bytes), parameter count 751,632,384 as stored, 596,049,920 unique
• Qwen3 model card: "DO NOT use greedy decoding" in thinking mode

API documentation (checked 2026-10-18):

• Anthropic Messages API reference: temperature not fully deterministic at 0.0; temperature, top_p and top_k restrictions for models released after Claude Opus 4.6; no logprobs parameter
• OpenAI Chat Completions reference: temperature, top_p, logprobs, top_logprobs, seed marked "Deprecated."

Measurements:

• The herb table and the ten herb samples: mlx-community/Qwen3-1.7B-4bit (Qwen3-1.7B with 4-bit weights) on Apple M1 Max, mlx 0.32.4, mlx-lm 0.32.2, measured for this course. The "tomato" result: Qwen3-0.6B on the same laptop's CPU in float32, same prompt, same three temperatures and ten seeded draws
• The fruit next-token table and the run-to-run 0.0 score difference: Qwen3-0.6B in bf16 on Apple M1 Max (MPS), transformers 5.20.1, torch 2.14.1, measured for this course on 2026-10-18
• The scripts in this lesson: Python 3.12; the herb script with mlx 0.32.4 and mlx-lm 0.32.2 on mlx-community/Qwen3-1.7B-4bit