Output as a contract

The output section Nora shipped, and the two beta failures from the video. The section in her instructions:

# Output
Reply only with JSON in this shape, and nothing else:
{"verdict": "checked",
 "corrections": [{"quote": "...", "fix": "...", "kind": "grammar", "explanation": "..."}],
 "rewrite": "...",
 "note": "..."}

What the browser console said when a teacher's long essays left the result panel empty:

SyntaxError: Unexpected end of JSON input

And the correction behind the "undefined" underline, from a Japanese learner's weekend in Umeda, which parsed without complaint:

{ "quote": ", the pancake", ".fix": ". The pancake", "kind": "punctuation", "explanation": "..." }

Ask, or Enforce?

Treat the prompt as an experiment and count two things: does the reply parse, and does it have the right shape? On Nora's 30 texts: 30 of 30 parse, 29 of 30 have the right shape. Today's model follows a request for JSON almost every time, and "almost" is the problem.

How stable is "almost"? The course's lab tried a different small task: extracting a product from five messy marketplace listings into a seven-field object, 20 runs each, with "Respond in JSON." in the prompt.

Before reading on, predict. The same prompt, on last year's small model. Out of twenty replies, how many parse?

None. Here are the three rows side by side, run from an Apple M1 Max laptop, November 2026:

Every reply from the older model came wrapped in a Markdown code fence. Here is reply 4, as the lab printed it:

#4: code fence + bad enum [('condition', 'open box')]: '```json\n{\n  "name": "V15 Detect cordless vacuum",\n  "brand": "Dyson",\n  "price": 1099.00,\n  "currency": "CAD",\n  "condition": "open box", ...

The reply starts with three backticks and the word json, so JSON.parse fails on the first character. And inside the fence, condition says "open box", which is not one of the three values the prompt allowed (new, used, refurbished). The model invented a fourth.

This table does not say that current models are bad at JSON. It says the same prompt gave 100 percent on one model and 0 percent on another. Asking gives you a probability that belongs to one model on one day. A feature needs a guarantee.

Structured Output

The third row is the guarantee. With structured output, the request carries a JSON Schema, and the provider only lets the model write tokens that keep the output inside that schema. A token that would start a code fence is not allowed. Neither is "open box" in the condition field, or a key called .fix where the schema says fix. The shape is guaranteed by the provider, not hoped for by you. The schema is the JSON Schema itself; in Nora's code it is a Zod object that produces it.

This course uses one picture for this, and only this one: the schema is a contract, the shape both sides agree on before the call. Nora's code promises to handle anything in that shape. The provider promises to return nothing outside it.

The contract has limits, and they get a lesson of their own, shape is not truth. The model can still break it: a refusal or a cut-off answer does not follow the schema. And the contract says nothing about truth.

What changed. Older code asked for "JSON mode" ({"type": "json_object"}), which guarantees valid JSON but not your shape: OpenAI's guide says "only Structured Outputs ensure schema adherence." Two old tricks for getting JSON from Claude now fail with a 400 on current models: ending the request with an assistant message that starts with { (prefill, removed from the 4.6 generation on), and forcing a tool call to get JSON (rejected on Opus 5.5, Sonnet 5.5 and Fable 5.1). Structured output is generally available on all three providers; use it.

Recap. Asking for JSON gives a probability that belongs to one model. A schema in the request gives a guarantee of shape. The schema is a contract on shape only: refusals and cut-offs can still break it, and it says nothing about truth.

Writing the Schema

Inkstep's schema has four fields at the top and four in each correction. In Zod:

// src/schema.ts: the shape both sides agree on before the call.
import { z } from "zod";
import { zodTextFormat } from "openai/helpers/zod";
 
export const Correction = z.object({
  quote: z.string(), // the learner's exact words, copied
  fix: z.string(),
  kind: z.enum(["grammar", "spelling", "word_choice", "punctuation"]),
  explanation: z.string(), // short, in the learner's language
});
 
export const CheckResult = z.object({
  verdict: z.enum(["checked", "not_english", "nothing_to_check"]),
  corrections: z.array(Correction).max(8),
  rewrite: z.string().nullable(),
  note: z.string(),
});
 
export type Correction = z.infer<typeof Correction>;
export type CheckResult = z.infer<typeof CheckResult>;
 
// sent as text.format on every call (and billed as input)
export const checkFormat = zodTextFormat(CheckResult, "check_result");

Each correction is one underline: quote is the learner's exact words, so the app can find them; fix is the replacement; kind is an enum, so the app can colour the underline and the model cannot invent a fifth kind; explanation is in the learner's language. At most eight, the house rule. The verdict, the nullable rewrite and the note get their reasons below. The z.infer lines give the rest of the code the same shape as TypeScript types.

The field order is on purpose. With a strict schema, the model writes the keys in schema order: the verdict first, because it decides what the app shows at all. That order matters again when the answer streams into the browser.

On the Wire

The request carries the schema in the text.format field, and the SDK's helper zodTextFormat turns the Zod object into a JSON Schema. Here is what it produces, run locally with openai 7.39.0 and zod 4.6.7 (no API call), trimmed:

{
  "type": "json_schema",
  "name": "check_result",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "verdict": { "type": "string", "enum": ["checked", "not_english", "nothing_to_check"] },
      "corrections": {
        "maxItems": 8,
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "quote": { "type": "string" },
            "fix": { "type": "string" },
            "kind": { "type": "string", "enum": ["grammar", "spelling", "word_choice", "punctuation"] },
            "explanation": { "type": "string" }
          },
          "required": ["quote", "fix", "kind", "explanation"],
          "additionalProperties": false
        }
      },
      "rewrite": { "type": ["string", "null"] },
      "note": { "type": "string" }
    },
    "required": ["verdict", "corrections", "rewrite", "note"],
    "additionalProperties": false
  }
}

"strict": true turns the guarantee on. Read the schema for three rules that OpenAI enforces in strict mode, because each one has bitten someone.

Every field is required. Both required lists name every property. A field that may be empty is a type that allows null: rewrite is ["string", "null"], which is what .nullable() produces. Use Zod's .optional() instead, and the SDK stops you before the request is sent. The lab tried it on a field called color:

zodTextFormat threw: Schema field at `properties/color` uses `.optional()` without `.nullable()` which is not supported by the API. See: https://platform.openai.com/docs/guides/structured-outputs?api-mode=responses#all-fields-must-be-required

Every object closes itself. "additionalProperties": false on every object says no other keys may appear. If you write the JSON Schema by hand and forget it, or leave a field out of required, the API rejects the request. These are the API's own 400 bodies, from the lab's product schema:

[no additionalProperties] 400 {"message": "Invalid schema for response_format 'product': In context=(), 'additionalProperties' is required to be supplied and to be false.", "type": "invalid_request_error", "param": "text.format.schema", "code": "invalid_json_schema"}
[optional field] 400 {"message": "Invalid schema for response_format 'product': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Missing 'price'.", "type": "invalid_request_error", "param": "text.format.schema", "code": "invalid_json_schema"}

The message still says response_format, the field's name on the older Chat Completions API. Error messages lag behind renames; read the param and code.

Categories are enums. kind and verdict are lists of allowed strings, so the output can only hold values the app knows how to draw. .max(8) became "maxItems": 8, a limit OpenAI enforces on arrays.

The output section Nora added comes out of the instructions again: the schema carries the shape. OpenAI lists this as a benefit, "Simpler prompting". What stays in the instructions is meaning: what counts as a correction, when to use each verdict.

What the Schema Costs

The schema is sent with every request, and the provider bills it as input tokens. In the lab, the seven-field product schema added about 82 input tokens to every call. Inkstep's schema, counted with the API's token counter, adds 111 to every check.

There is also a one-time cost. OpenAI's structured outputs guide says "the first request you make with any schema will have additional latency as our API processes the schema, but subsequent requests with the same schema will not." The course did not time it; change the schema, and expect the next request to be slower once.

Recap. Write the schema in Zod and send it with zodTextFormat in text.format. In strict mode every field is required (use .nullable() for "may be empty"), every object sets additionalProperties: false, and categories are enums. The schema is billed as input on every call.

Before going on, use an earlier rule on this new cost. The schema adds 111 tokens to every check. In which usage field will you see them?

In input_tokens. The schema is part of what the model reads, so it is counted and billed as input, at the input price, on every check.

A Way to Say "I Can't"

The schema now forces every answer into Inkstep's shape. Before reading on, predict. A learner pastes a paragraph in Spanish. What comes back?

Something in the shape, because nothing else is allowed. Picture the schema without a verdict field: corrections, a rewrite and a note, all required. The model must fill them, so it may correct the Spanish as if it were bad English, or invent corrections for a learner who typed only "hello". OpenAI's guide warns that the model "will always try to adhere to the provided schema", which "can result in" made-up content "if the input is completely unrelated to the schema."

The fix is to make "I can't" a valid answer inside the contract. That is what two of Inkstep's fields are for:

• verdict has three values. checked: the text is English and was checked. not_english: it is not English. nothing_to_check: it is too short to check, like "hello".
• rewrite is nullable. When the verdict is not checked, there is nothing to rewrite, and null says so.

The instructions say when to use each one:

# Verdict
 
- `checked`: the text is English writing you can check, with or without mistakes.
- `not_english`: most of the text is not in English. Return no corrections and
  `rewrite: null`. The note says, in the explanation language, that the app
  checks writing in English.
- `nothing_to_check`: the text is too short to check (for example a greeting or a
  single word). Return no corrections and `rewrite: null`. The note asks the
  learner, in the explanation language, to write a few sentences about the task.

In the lab, a Spanish paragraph got not_english in 3 of 3 runs, and "hello" got nothing_to_check in 3 of 3, each with no corrections and a note saying what to do next. The app can show "This doesn't look like English" instead of fake underlines.

Two Fields Nora Does Not Add

A confidence field. You might expect a confidence score to tell you which answers are wrong: show the corrections above 0.8, hide the rest. That is wrong. Models state their confidence badly: a study at ICLR 2024 (Xiong et al.) found stated confidence "overconfident, potentially imitating human patterns of expressing confidence", and little better than chance at telling right answers from wrong. Give the model a way to say "I can't", and check the answer yourself, against data you trust. The next lesson, shape is not truth, builds that check.

A reasoning field. Older tutorials put one first in the schema, so the model would reason before answering. Inkstep's model already reasons before any visible output, so the field pays for the same work twice. On Anthropic's current models it can also cause a billed refusal: its docs list "A reasoning, thinking, or trace field in JSON output or in a tool input" as a cause of the reasoning_extraction refusal.

Recap. A strict schema forces an answer even when there is none, so put the way out inside the schema: a verdict enum and a nullable rewrite, with an instruction for each case. Do not add confidence or reasoning fields.

The Same Idea on Claude and Gemini

The schema, the way out, and the two ways the contract breaks, on each provider. From the docs, checked 2026-11-03:

The rows in the middle are the reason Nora also checks the shape on her own side, with the shape check in her code. One Zod schema, sent to three providers, is enforced three different ways. Your own validation is the one that is the same everywhere.

The Takeaway

Back to the two beta failures. The "undefined" underline was a misspelled key in valid JSON, and the strict schema makes that impossible.

1. Asking for JSON gives a probability that belongs to one model (200 of 200 on today's small model, 0 of 20 on last year's). A strict schema gives a guarantee of shape.
2. Write the schema in Zod and send it with zodTextFormat: every field required (.nullable() for empty), every object closed, categories as enums. It is billed as input on every call.
3. Put the way to say "I can't" inside the schema: a verdict enum and a nullable rewrite. Skip confidence and reasoning fields.

The one sentence to keep: a strict schema guarantees the shape of every answer, and a verdict field lets the model say that it cannot.

An open question to carry forward: a teacher's long essays still leave the result panel empty. What does a strict schema do when the answer is cut off? That is the next step, shape is not truth.

References

Provider documentation (checked 2026-11-03):

• Structured outputs (OpenAI): strict mode rules (all fields required, additionalProperties: false, supported keywords), "the first request you make with any schema will have additional latency", keys in schema order, the refusal content part, "Structured Outputs can still contain mistakes", "Simpler prompting", "only Structured Outputs ensure schema adherence", and the warning that the model "will always try to adhere to the provided schema"
• Structured outputs (Anthropic): output_config.format, optional fields and their limits, unsupported number and string limits, SDK helpers that move constraints into descriptions
• Refusals and fallback (Anthropic): the reasoning_extraction category, "A reasoning, thinking, or trace field in JSON output or in a tool input", and billing of refusals before output
• Prompting best practices (Anthropic): prefill unsupported from the 4.6 generation on
• Errors (Anthropic): forced tool_choice returns 400 on Opus 5.5, Sonnet 5.5 and Fable 5.1
• Structured output (Google): response_format, responseJsonSchema, constraints that are "automatically" stripped, and "Implement robust error handling for schema-compliant but semantically incorrect outputs"

Papers:

• Xiong et al., "Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs", ICLR 2024, arXiv:2306.13063: verbalized confidence is "overconfident, potentially imitating human patterns of expressing confidence"; failure-prediction AUROC between 0.522 and 0.605

Lab runs (Apple M1 Max, Node 24.14.1, openai 7.39.0, zod 4.6.7, 2026-11-03 and 2026-11-04):

• Prompt-only JSON (check-v1 with the output section above, no schema) on the 30 check-set texts, gpt-6.1-sol at medium, 1 run each: 30 of 30 parsed, 29 of 30 had the right shape (ja-02: a correction with the key .fix), no code fences
• "Respond in JSON." against a strict schema on five marketplace listings, 20 runs each: gpt-6-luna at effort none 20 of 20 plus 180 of 180, gpt-4o-mini 0 of 20 (every reply in a code fence, one with the invented value "open box"), strict 20 of 20 on both; about +82 input tokens per call for that schema; "open box (returned unused)" forced to used in every strict run
• The 400 invalid_json_schema bodies and zodTextFormat rejecting .optional(); the JSON Schema for CheckResult and the Zod errors, run locally
• Inkstep's schema size: 111 input tokens, client.responses.inputTokens.count
• The verdicts: a Spanish paragraph (not_english) and "hello" (nothing_to_check), 3 of 3 runs each, gpt-6.1-sol at medium

Pinned packages: openai 7.39.0, zod 4.6.7, Node 24.14.1.