Shape is not truth

Is It Complete?

The long essay from the video, as the lab logged it (1,516 words, the mid-size model at medium, the strict schema, the cap of 1,500):

essay 1516 words, 52.9 s
status incomplete, incomplete_details {"reason":"max_output_tokens"}
usage {"input_tokens":3937, ... "output_tokens":1500,"output_tokens_details":{"reasoning_tokens":1500}}
output items ["reasoning","reasoning","reasoning"]
JSON.parse: Unexpected end of JSON input; parseResult: null
output_text length 0 chars; last 160: ""

Every token went to reasoning, so the text is empty, and JSON.parse("") throws exactly the error from the console.

When the cap lands later, inside the JSON, the text ends mid-string instead. The lab's other task, extracting a product from a marketplace listing, at a cap of 30 tokens, small model at effort none (Python SDK):

[strict json] status=incomplete incomplete_details=IncompleteDetails(reason='max_output_tokens') usage=30 out (0 reasoning)
   text='{"name":"Dyson V15 Detect Cordless Vacuum (Open Box)","brand":"Dyson","features":["Laser'
   json.loads: Unterminated string starting at: line 1 column 83 (char 82)

Either way, the response said so before anyone parsed anything: status is "incomplete", and the reason is "max_output_tokens". Structured output does not save you from the stop reason. Read it first. Anthropic gives the rule for its own API in one line, and it holds on every provider:

"Treat any response whose stop_reason is "max_tokens" as failed, even if its text holds valid JSON"

The Refusal

The other way the contract breaks is a refusal. A refusal, as the first call noted, is a 200 with a refusal content part:

{ "type": "refusal", "refusal": "I'm sorry, I cannot assist with that request." }

That shape is from OpenAI's guide; the lab could not provoke a refusal with a harmless test. Refusals are rare on a task like Inkstep's, but a refusal's text does not follow your schema, so the code checks for it before parsing.

The Gate

Put together, Nora's code checks every response in a fixed order before it trusts anything in it:

1. Status. Incomplete means cut off (or, rarely, stopped by a content filter). Stop.
2. Refusal. A refusal content part means the model declined. Stop.
3. Parse. JSON.parse the text.
4. Shape check. Validate the parsed value with Zod.

// src/validate.ts: why it stopped, then the shape, then the meaning.
import type { Response } from "openai/resources/responses/responses";
import { CheckResult, type Correction } from "./schema.ts";
import type { CheckInput } from "./prompt.ts";
 
export type Gate =
  | { ok: true; text: string }
  | { ok: false; reason: "cut_off" | "refused" | "unavailable" };
 
export function gate(response: Response): Gate {
  if (response.status === "incomplete") {
    const why = response.incomplete_details?.reason;
    return { ok: false, reason: why === "content_filter" ? "refused" : "cut_off" };
  }
  if (response.status !== "completed") return { ok: false, reason: "unavailable" };
  const content = response.output.flatMap((item) => (item.type === "message" ? item.content : []));
  if (content.some((part) => part.type === "refusal")) return { ok: false, reason: "refused" };
  return { ok: true, text: response.output_text };
}
 
export function parseResult(text: string): CheckResult | null {
  let json: unknown;
  try {
    json = JSON.parse(text);
  } catch {
    return null; // not JSON at all
  }
  const shape = CheckResult.safeParse(json); // the shape check
  return shape.success ? shape.data : null;
}

gate reads the status first; any status other than completed is a failure. Then it collects the content parts of every message item, by type, and fails on a refusal. Only then does it hand back the text. parseResult parses it and runs the shape check: Zod's safeParse against the same schema that was sent.

Why check the shape again, when the provider guarantees it? Because the guarantee has edges. A provider may quietly skip a rule, as the provider table showed. And the day a request goes out without the schema, as Nora's beta did, the shape check still holds. Here is what it says about the Umeda reply from Nora's beta, the one with the key .fix in place of fix, and about a result with nine corrections:

message  Invalid input: expected string, received undefined
path     ["corrections", 3, "fix"]
 
message  Too big: expected array to have <=8 items
path     ["corrections"]

The Fix for Long Essays

The gate stops the crash. The learner still gets no feedback on a long essay. What should change?

The tempting fix is a bigger cap. It is the wrong one. A bigger cap is a bigger worst-case bill on every check, and under the provider's rate-limit rule it can count against the limit even on checks that never use it, as failures shows. And it does not bound the problem: the longer the essay, the more the model reasons and the longer the rewrite, which is the whole text again.

So the fix is on the input side. The text box accepts at most 400 words, with a friendly message when the learner goes over. Does a 400-word check fit in 1,500 output tokens? The lab measured the closest cases it had:

THE LARGEST CHECK THE TEXT BOX ALLOWS (mid-size model, medium effort)
  a 400-word text                     learner block 515 tokens (counted)
  the two longest check-set texts (378 and 379 words), 12 checks:
    status                            completed, 12 of 12
    output tokens                     1,173 to 1,468 (median 1,270)
    of which reasoning                219 to 516
  the cap                             1,500
  room left on the largest            1,500 - 1,468 = 32 tokens

It fits, with little room to spare. Reasoning varies from run to run, so a rare check near the limit can still be cut off, and the gate catches it with a clear message instead of a crash. The cap stays at 1,500. The limit in the text box is the fix; the gate is the safety net under it.

Recap. A strict schema does not make the model finish. Gate every response before parsing: status first, then refusal, then parse, then the shape check with Zod. Fix cut-off answers by limiting the input that makes them, not by raising the cap.

Is It True?

Here is the prediction to pause on. Structured output is on. Can a correction still point at words the learner never wrote?

Take a moment before you read on.

Yes. The schema says quote is a string. Any string. It cannot say "a string copied exactly from the learner's text", because JSON Schema has no way to say that. Here is a real correction from the lab, with the strict schema on, from last year's small model (gpt-4o-mini), for the Pune paragraph:

{ "quote": "there was too much crowd", "fix": "there was too much crowds", "kind": "grammar", "explanation": "..." }

The learner wrote "There was too much crowd", with a capital T. The model dropped the capital while copying, so the app finds nothing and cannot place the underline. The JSON is perfect. (The fix is wrong too; no check in this lesson can see that, and the check set later in the course is for it.)

How often does this happen? On today's models, rarely. In the lab, the mid-size model wrote 386 corrections across 90 checks and every quote matched exactly once. Across every run on today's two models, 2 of 3,164 corrections missed. On last year's small model, 12 of 310 did, about 1 in 25.

You might expect structured output to mean the content is right. It does not, and the providers say so. OpenAI: "Structured Outputs can still contain mistakes." Gemini asks you to "Implement robust error handling for schema-compliant but semantically incorrect outputs." So Nora's next check is insurance: against the older or cheaper model she may run tomorrow, and against a model that changes under her.

The Meaning Check

A meaning check tests the result against Inkstep's own data: the learner's text and the house rules. The textbook name is semantic validation. For one correction, Nora checks four things, each cheap and exact:

1. The quote appears exactly once in the learner's text. Zero times means the app cannot place the underline. Twice means it cannot know which one to underline.
2. The fix is different from the quote. A correction that changes nothing is noise.
3. For Hindi and Japanese, the explanation is written in that script. An explanation in English for a learner who chose Hindi has failed its job.
4. No correction changes a spelling the house rules accept, such as "colour" or "travelling".

// src/validate.ts (continued)
const SCRIPTS: Record<string, RegExp> = {
  Hindi: /\p{Script=Devanagari}/u,
  Japanese: /[\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Han}]/u,
};
const ACCEPTED_SPELLINGS = ["colour", "favourite", "centre", "organise", "travelled", "travelling", "practise"];
 
export function meaningProblems(c: Correction, input: CheckInput): string[] {
  const problems: string[] = [];
  const count = input.text.split(c.quote).length - 1;
  if (count !== 1) problems.push(`"${c.quote}" appears ${count} times in the text`);
  if (c.fix === c.quote) problems.push(`"${c.quote}" has a fix identical to the quote`);
  const script = SCRIPTS[input.explanationLanguage];
  if (script && !script.test(c.explanation)) {
    problems.push(`the explanation of "${c.quote}" is not in ${input.explanationLanguage}`);
  }
  const touched = ACCEPTED_SPELLINGS.find((w) => c.quote.includes(w) && !c.fix.includes(w));
  if (touched) problems.push(`"${c.quote}" changes "${touched}", which the house rules accept`);
  return problems;
}

split(c.quote).length - 1 counts how many times the quote occurs, case and all. The script test uses Unicode script classes and asks only whether the explanation contains that script at all, which catches the common failure, a whole explanation in English. Portuguese, Spanish and Indonesian use the Latin alphabet, so they get no script test. Each problem is a sentence in plain words, which matters in a moment.

Which checks fire in practice? In the lab, the missing quotes had three causes: the model changed a capital letter (12 of the 14 cases across all runs, usually a sentence's first word written in lower case), edited the words while copying ("let us work from home" for "lets us work from home"), or left a word out ("we reached" for "we all reached"). On today's mid-size model, the check that fired was the fourth: 3 of 386 corrections changed "travelling" or "favourite". On last year's small model, 15 of the 36 checks for Hindi and Japanese learners had explanations in English (44 explanations in all), and the script test caught every one.

What to Do With a Problem

A meaning check that fails needs a decision. Nora's rule:

• Drop the correction. The learner sees one underline fewer. A missing underline is a small loss; a wrong underline teaches the learner something false and makes them doubt the rest.
• If more than two corrections in one check are dropped, the check as a whole is in doubt. Re-ask once: send the request again with the list of problems added, and ask for the whole result again.
• Never loop. After the re-ask, drop whatever still fails and return the rest.

A re-ask is a different request: the same instructions and learner block, plus a note listing what was wrong. Here is a real one, from a Japanese learner's check on last year's small model, where all three explanations came back in English:

Your previous answer had these problems:
- the explanation of "in a apartment" is not in Japanese
- the explanation of "is office worker" is not in Japanese
- the explanation of "goes to walk" is not in Japanese
Return the whole result again. Copy every quote exactly from learner_text.

The second answer had all three in Japanese. The plain-words problem list went straight into the re-ask. Keep the two words apart. A retry sends the same request again, because the request did not reach the model or the provider failed; failures owns retries. A re-ask sends a different request, because the answer came back wrong. Calling it a retry hides the fact that it changes what the model reads.

Every safeguard has a price, and someone pays it. Here the price is a whole second check. On the mid-size model, with the course's measured medians:

ONE CHECK (mid-size model, medium effort)
  input    2,294 tokens   $0.0005471   (most of it billed at the cached price,
                                        which the course explains with the bill)
  output     510 tokens   $0.0051000   (335 visible + 175 reasoning, x $10.00 / 1,000,000)
  total                   $0.0056471   about half a cent
 
ONE RE-ASK          one more check, about $0.0056 (its input is a little longer)
 
IF 3 PERCENT OF CHECKS WERE RE-ASKED
  extra per check   0.03 x $0.0056471 = $0.00017   3 percent more on the bill

The measured rates are lower. In the lab, the mid-size model needed no re-ask in 90 checks. Last year's small model needed one in 9 of 90 checks, and one re-ask fixed all 9, at about $0.00037 each on that model. The learner who waits for that second answer pays too. That is why the threshold is "more than two dropped", not "any": one bad quote is cheaper to drop than to re-ask.

Recap. The schema guarantees shape, not truth. Check meaning against your own data: every quote exactly once in the text, fix different from quote, the explanation in the right script, accepted spellings untouched. Drop what fails; re-ask once if more than two fail; never loop.

What the Program Gets Back

All of this ends in one function, runCheck, in src/check.ts. It is what the rest of Inkstep calls. Its return type is the most important line in the file:

// src/check.ts (excerpt)
export type CheckOutcome =
  | { ok: true; result: CheckResult; dropped: number }
  | { ok: false; reason: "cut_off" | "refused" | "invalid" | "unavailable" };
 
export type ModelClient = {
  create(request: ResponseCreateParamsNonStreaming): Promise<Response>;
};

A check outcome is either ok, with a result that passed every check and the number of corrections dropped, or a failure with one named reason: cut_off, refused, invalid (did not parse or failed the shape check) or unavailable (the call itself failed). The app can only draw underlines from an ok outcome, because only an ok outcome has a result. TypeScript enforces that: code that reads outcome.result without first checking outcome.ok does not compile.

runCheck takes the model as a parameter, a ModelClient. In production it wraps the SDK: { create: (request) => client.responses.create(request) }. Passing it in, instead of creating the client inside, is what lets a test hand runCheck a stand-in later in the course.

Here is the rest:

// src/check.ts (continued)
export function checkRequest(input: CheckInput): ResponseCreateParamsNonStreaming {
  return {
    model: MODEL,
    reasoning: { effort: EFFORT },
    max_output_tokens: MAX_OUTPUT_TOKENS,
    store: false, // learner writing is personal; Inkstep never needs it back
    text: { format: zodTextFormat(CheckResult, "check_result") },
    ...buildCheckRequest(input),
  };
}
 
export function reaskNote(problems: string[]): string {
  return [
    "Your previous answer had these problems:",
    ...problems.map((p) => "- " + p),
    "Return the whole result again. Copy every quote exactly from learner_text.",
  ].join("\n");
}
 
export async function runCheck(model: ModelClient, input: CheckInput): Promise<CheckOutcome> {
  let request = checkRequest(input);
  for (let attempt = 1; ; attempt++) {
    let response: Response;
    try {
      response = await model.create(request);
    } catch {
      return { ok: false, reason: "unavailable" }; // HTTP errors get a real policy later
    }
    const gated = gate(response); // 1. why did it stop?
    if (!gated.ok) return gated;
    const result = parseResult(gated.text); // 2. the parse and the shape check
    if (!result) return { ok: false, reason: "invalid" };
    const problems = result.corrections.map((c) => meaningProblems(c, input)); // 3. the meaning check
    const kept = result.corrections.filter((_, i) => problems[i].length === 0);
    const dropped = result.corrections.length - kept.length;
    if (dropped <= 2 || attempt === 2) {
      return { ok: true, result: { ...result, corrections: kept }, dropped };
    }
    // a re-ask is a different request (the problems added); it happens once, never in a loop
    request = { ...request, input: request.input + "\n\n" + reaskNote(problems.flat()) };
  }
}

checkRequest is the request from the prompt step plus one line: the schema in text.format. runCheck walks the checks in order. A failed call is unavailable for now; failures replaces that catch with a real policy. A failed gate returns its own reason, a failed parse or shape check is invalid, and the meaning check drops what fails. More than two drops on the first attempt lead to one re-ask, with reaskNote added to the end of the input; on the second attempt, whatever is left is returned. The dropped count goes back with the result, and later becomes one of the numbers Inkstep watches in production.

On the path of one check, Nora's server has its second box now. buildCheckRequest builds the request; validate stands between the provider's response and the learner's screen, and only an ok outcome passes it.

Check today. On 2026-11-03, OpenAI's strict mode required every field and additionalProperties: false, supported maxItems and number limits, and did not support string length limits. Anthropic allowed optional fields (24 across a request) but no number limits; Gemini enforced number limits and silently removed string lengths. These rules change between API versions. Read the current list before you design a schema: https://developers.openai.com/api/docs/guides/structured-outputs

Recap. runCheck returns a CheckOutcome: typed corrections or a named failure, never something in between. It takes the model as a parameter, assembles the request in one place, gates, parses, checks shape and meaning, drops, and re-asks at most once.

The Takeaway

Back to the empty panel. The long essays were cut-off answers, parsed without reading why they stopped: the model spent the whole cap on reasoning. The gate now catches them, and the 400-word text box prevents them. The meaning check covers what the schema and the gate cannot see: quotes the learner never wrote, explanations in the wrong script, accepted spellings changed.

1. Gate before parsing: status, refusal, parse, shape check. Fix cut-offs on the input side, not with a bigger cap.
2. Check meaning against your own data. Drop what fails, re-ask once if more than two fail, never loop. A re-ask is a different request, not a retry, and it costs a second check.
3. runCheck returns typed corrections or a named failure, and takes the model as a parameter.

The one sentence to keep: a schema guarantees the shape of the output, not its truth: check why the model stopped, then the shape, then the meaning against your own data.

An open question to carry forward: the result is right now, but the learner stares at a spinner for about twelve seconds while the whole answer is written. Can the underlines appear while the model is still writing? That is streaming, one underline at a time.

References

Provider documentation (checked 2026-11-03):

• Structured outputs (OpenAI): strict mode rules (all fields required, additionalProperties: false, supported keywords), "the first request you make with any schema will have additional latency", keys in schema order, the refusal content part, "Structured Outputs can still contain mistakes", "Simpler prompting", "only Structured Outputs ensure schema adherence", and the warning that the model "will always try to adhere to the provided schema"
• Reasoning models (OpenAI): a cut-off "might occur before any visible output tokens are produced, meaning you could incur costs for input and reasoning tokens without receiving a visible response"
• Structured outputs (Anthropic): output_config.format, optional fields and their limits, unsupported number and string limits, SDK helpers that move constraints into descriptions
• Prompting Claude Sonnet 5.5 (Anthropic): "Treat any response whose stop_reason is "max_tokens" as failed, even if its text holds valid JSON"
• Structured output (Google): response_format, responseJsonSchema, constraints that are "automatically" stripped, and "Implement robust error handling for schema-compliant but semantically incorrect outputs"

Lab runs (Apple M1 Max, Node 24.14.1, openai 7.39.0, zod 4.6.7, 2026-11-03 and 2026-11-04):

• The 400 invalid_json_schema bodies and zodTextFormat rejecting .optional(); the JSON Schema for CheckResult and the Zod errors, run locally
• The 1,516-word essay as a check-v1 request, gpt-6.1-sol at medium, cap 1,500: incomplete, 1,500 of 1,500 output tokens reasoning, empty text, 52.9 s; the cut-off at max_output_tokens: 30 in strict JSON, gpt-6-luna at effort none, Python SDK
• The two longest check-set texts (378 and 379 words), 6 runs each on gpt-6.1-sol at medium: 1,173 to 1,468 output tokens, all completed; a 400-word text's learner block, 515 tokens
• Missing quotes and re-asks, every check-set run re-checked offline: gpt-6.1-sol at medium 0 of 386 corrections, 0 of 90 checks re-asked, 3 of 386 dropped for an accepted spelling; today's two models 2 of 3,164; gpt-4o-mini 12 of 310 (10 a changed capital letter, 1 edited while copying, 1 a word left out), 44 explanations in English for Hindi and Japanese learners (15 of 36 checks), 9 of 90 checks re-asked and all 9 fixed by one re-ask, $0.0033 for the 9

Pinned packages: openai 7.39.0, zod 4.6.7, Node 24.14.1.