Counting the work, forcing the shape

Here is the model file for Llama 3.1 8B, an open model whose size anyone can read: about eight billion parameters, and 16 GB of weights. Each pass runs every layer once and ends with one score for every token in the vocabulary. A Cookvale answer is 300 of those passes, one after another. This lesson counts what that work costs, then shows how an API can promise that a tool call is valid JSON, and what that promise never covers.

What One Token Costs to Compute

How much arithmetic is one pass? Count for Llama 3.1 8B. In a matrix multiply, each weight is used once per token, for one multiply and one add. So the rule is about 2 arithmetic operations per parameter per token. Kaplan and others give the same rule in their 2020 scaling-laws paper, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication".

The embedding table is the exception: a lookup costs no arithmetic.

all parameters                         8,030,261,248
minus the embedding table               - 525,336,576   (a lookup)
parameters used in arithmetic          7,504,924,672
 
x 2 operations each                   15,009,849,344
                                      about 15.0 billion operations per token

Fifteen billion operations for every token, in every pass. You will also see these counted as FLOPs (floating-point operations).

Attention Adds Work That Grows

That 15.0 billion is the weights' share. Attention adds more, and its share depends on how many tokens came before. For each earlier token, the new token compares its query with a key and mixes in a value:

compare with one earlier key      4,096 multiplies + 4,096 adds   =   8,192
mix in that token's value         4,096 multiplies + 4,096 adds   =   8,192
per earlier token, per layer                                      =  16,384
x 32 layers                                                       = 524,288

So each earlier token adds 524,288 operations. For the last token of Cookvale's 12,060-token turn 1:

attention    524,288 x 12,060              = 6,322,913,280   about 6.3 billion
weights                                     = 15,009,849,344  about 15.0 billion
attention as a share of the weights' work    6.3 / 15.0       = 42 percent

When does attention catch up with the weights?

15,009,849,344 / 524,288 = 28,629   about 28,600 tokens

Below about 28,600 tokens of context, the weights do most of the arithmetic on this model. Above it, attention does. Cookvale's chats stay well below that point. At 131,072 tokens, the longest Llama 3.1 8B accepts, attention is more than four times the weights' share for that token. This is why very long prompts get expensive faster than their token count suggests.

One Answer Is Many Passes

Now the loop. A Cookvale answer averages 300 tokens. Each token needs its own pass, and each pass runs all 32 layers. Leviathan and others state it in their 2023 paper on speculative decoding: "decoding K tokens takes K serial runs of the model." Serial means one after another: pass 2 cannot start until pass 1 has chosen its token, because pass 2 reads that token.

passes for a 300-token answer       300
layer runs                          300 x 32 = 9,600
 
arithmetic, weights                 300 x 15.0 billion             = 4.50 trillion
arithmetic, attention               300 x 524,288 x about 12,210
                                    (average tokens before each)   = 1.92 trillion
total for the answer                                               = about 6.4 trillion operations

Is 6.4 trillion a lot? An NVIDIA H100 lists 989 trillion operations per second for these numbers:

6.4 trillion / 989 trillion per second = 0.0065 s, about 6.5 thousandths of a second

Yet the pause, the rush and the bill put these 300 tokens at about 1.6 seconds on that GPU. Something other than arithmetic sets the speed of the 300 passes. Prefill and decode finds it.

Each new token costs about two operations per parameter plus a little more for every earlier token, and an answer of 300 tokens is 300 of those passes, one after another.

Forcing the Shape of the Output

The sampler has one more power. It can refuse tokens.

Cookvale's assistant has a tool called add_to_shopping_list. When a user says "put basil on my list", the model does not call anything itself. It writes the tool's arguments as text, token by token, and the app reads that text as JSON. Here is the tool, in Anthropic's tool format, with strict tool use turned on:

{
  "name": "add_to_shopping_list",
  "description": "Add ingredients to the user's shopping list.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "items": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "quantity": { "type": "number" },
            "unit": {
              "type": "string",
              "enum": ["g", "kg", "ml", "l", "tsp", "tbsp", "piece", "bunch"]
            }
          },
          "required": ["name", "quantity", "unit"],
          "additionalProperties": false
        }
      }
    },
    "required": ["items"],
    "additionalProperties": false
  }
}

input_schema is a JSON Schema: items is a list, each item has a name (text), a quantity (a number) and a unit that must be one of eight values, and nothing else is allowed.

Without help, nothing stops the model from writing something else: every vocabulary token has a score at every step. A comma can end a list, or the quantity can come out as the word "a". OpenAI's launch post for this feature says it plainly: "they are generally free to sample a curly brace token at any time, even when that would not produce valid JSON."

The Mask

The fix is constrained decoding. Before the sampler draws, a grammar built from the schema checks every token in the vocabulary. Every token that would break the schema gets its score set to minus infinity. After softmax, a score of minus infinity has a probability of exactly 0: its gap to every other token is infinite. So only legal tokens can be drawn. The model and its pass are unchanged. Only the menu shrinks.

That is the "grammar mask" in the sampler order from the previous section. Follow it through the call for "put basil on my list":

written so far                                  the grammar allows next
(nothing)                                       an opening brace (or a space)
{"items": [{"name": "                           any text, or a quote to end it
{"items": [{"name": "basil", "quantity":        a space or the start of a number
{"items": [{"name": "basil", "quantity": 1,
  "unit": "                                     only tokens that begin one of:
                                                g kg ml l tsp tbsp piece bunch
{"items": [{"name": "basil", "quantity": 1,
  "unit": "bunch"}]                             a closing brace, then the end

Suppose that after "quantity": the highest-scoring token is " followed by a, as if the model wanted to write "a bunch". Both are illegal there. The mask sets their scores to minus infinity, and the sampler draws from the tokens that start a number. The model writes 1.

OpenAI describes its own implementation in the same terms: "after every token, our inference engine will determine which tokens are valid to be produced next based on the previously generated tokens and the rules within the grammar that indicate which tokens are valid next. We then use this list of tokens to mask the next sampling step, which effectively lowers the probability of invalid tokens to 0." Anthropic's documentation says strict tool use exists "To enforce JSON Schema compliance on tool inputs with grammar-constrained sampling".

Why bother, if a good model writes valid JSON most of the time? OpenAI's launch post gives the numbers on its own schema benchmark (checked 2026-10-18):

Training got the model to 93 percent. In the post's words, that "still did not meet the reliability that developers need", so they "took a deterministic, engineering-based approach to constrain the model's outputs to achieve 100% reliability."

Older JSON mode is weaker. OpenAI's docs: "JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors." It allows any valid JSON. A schema mask allows only your shape.

Tokens Do Not Line Up With Grammar

A grammar works in characters; the model writes tokens. In the o200k_base tokenizer, {" is a single token, an opening brace and a quote together, and so are ": and "}. Each crosses a grammar boundary. So at every step the engine must check the full text of each candidate token against the grammar's current state, for all 150,000 to 200,000 tokens. Open-source engines such as XGrammar and llguidance make this fast with precomputed masks; llguidance reports "on the order of 50μs of CPU time per token (for 128k tokenizer)".

What It Costs, and What It Cannot Do

The mask is not free:

• A first-call delay. The schema is compiled into a grammar first. OpenAI's launch post: "Typical schemas take under 10 seconds to process on the first request, but more complex schemas may take up to a minute." Anthropic: "Compiled grammars are cached for 24 hours from last use".
• Escape cases. Anthropic's docs list two ways a response can still break the schema: a refusal (stop_reason: "refusal") and running out of tokens (stop_reason: "max_tokens"). A list cut off at the token limit is not valid JSON, mask or not.
• Schema features. Not every JSON Schema rule can become a grammar. Anthropic's list of unsupported features includes "Numerical constraints (such as minimum, maximum, multipleOf)".

Now a prediction. Strict tool use is on for add_to_shopping_list. Can the model still put a wrong value in a valid field?

Yes. {"name": "oregano", "quantity": 1, "unit": "bunch"} is valid when the user asked for basil. So is a quantity of 100. The mask checks shape, not truth: among the legal tokens, the sampler still picks by the model's scores. Your code still checks that 100 bunches of basil makes sense.

Whether constraints lower quality is argued. Tam and others reported in 2024 that "stricter format constraints generally lead to greater performance degradation in reasoning tasks"; the makers of the Outlines library replied that the evaluation was flawed. Either way, when an answer needs reasoning, put a free-text reasoning field before the answer field in the schema.

Constrained decoding sets the score of every illegal token to minus infinity before the draw, so the output always has the schema's shape, but the values inside it are still the model's choice.

The Takeaway

1. Each token costs about 2 operations per parameter, 15.0 billion for Llama 3.1 8B, plus 524,288 for every earlier token. A 300-token answer is 300 passes, one after another.
2. Constrained decoding sets illegal tokens to minus infinity before the draw: the output always has the schema's shape, but the values are still the model's choice.

The one sentence to keep: every token costs about two operations for every weight, and a grammar can force the shape of the output, but never its values.

An open question for the next lesson: each pass uses every weight of the model, all sixteen gigabytes of them for Llama 3.1 8B. The arithmetic of a whole 300-token answer would take a GPU a few thousandths of a second, yet the answer takes more than a second to stream. How fast can the GPU's memory deliver those sixteen gigabytes, again and again, once for every token? That is the next step, prefill and decode.

References

Papers:

• Kaplan et al., "Scaling Laws for Neural Language Models", 2020: about 2 operations per parameter per token, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication"
• Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023: "decoding K tokens takes K serial runs of the model"
• Tam et al., "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models", 2024; reply by Will Kurt, ".txt: Say What You Mean"
• Dong et al., "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", 2024

Model files on Hugging Face (checked 2026-10-18):

• Llama 3.1 8B Instruct config.json, read from the unsloth/Llama-3.1-8B-Instruct mirror (Meta's repository is gated); parameter count 8,030,261,248

API documentation (checked 2026-10-18):

• Anthropic structured outputs: constrained decoding, strict tool use, grammar caching for 24 hours, refusal and max_tokens cases, unsupported numerical constraints
• OpenAI Chat Completions reference: temperature, top_p, logprobs, top_logprobs, seed marked "Deprecated."
• OpenAI Structured Outputs guide: JSON mode vs structured outputs
• OpenAI, "Introducing Structured Outputs in the API", 2024-08-06: the token mask, the context-free grammar, 93 vs 100 percent, first-request latency
• llguidance README: about 50μs of CPU time per token

Measurements:

• The {", ": and "} tokens: tiktoken 0.14.0, o200k_base, measured for this course on 2026-10-18