Why does it break my JSON?
Structured output (constrained decoding) · Builder · 5 min · sampling, llm · after Next-token prediction, Temperature and sampling
A model writes JSON one token at a time, so nothing stops a token that breaks it. Structured output modes give the API your schema and block every token that cannot fit; values still need checking.
1 Guess first
2 Look
One token at a time
1 / 4Prompt: “Return the user as JSON with name and age.”
Sure! Here is the JSON: ```json {"name": "Ada", "age": thirty-six} ```JSON.parse → SyntaxError
illustrative failure, not a recorded output
Asked for JSON, it adds a friendly sentence and writes thirty-six. The parser gives up.
Each token is a choice among likely options, some of which break JSON; with a schema, the API removes those options before the pick, so the format holds but the values still need checking.
Constrained decoding means the API knows your schema and, at every step, allows only tokens that can still lead to valid output. Anthropic and Gemini describe their structured outputs this way. OpenAI separates JSON mode (valid JSON, any shape) from Structured Outputs, which follow your schema.
4 You are the model
You are the filter. The schema is fixed; at each step, pick the one next token it allows.
schema: {name: string, age: integer, plan: "free" | "pro"} · no extra keys
(empty)▍
candidate tokens illustrative
You are the filter: which next token does the schema allow?5 Check
6 Say it back
7 Mission · 5 min
With a free Gemini API key, ask Gemini 3.8 Flash to extract a recipe twice: once with a JSON schema in response_format (as in Gemini’s structured-output docs), once just asking for JSON in the prompt. Parse both. Then add maxLength to a field and check whether it is enforced.
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Sources
- https://ai.google.dev/gemini-api/docs/structured-output (updated 2026-10-20, checked 2026-10-21): response_format with mime_type application/json and a schema; examples use gemini-3.8-flash; “While output is syntactically correct JSON, always validate values in your application”; minLength, maxLength, minProperties, maxProperties “not enforced during constrained decoding and are ignored by the server”
- https://developers.openai.com/api/docs/guides/structured-outputs (checked 2026-10-21; no update date): “only Structured Outputs ensure schema adherence” (not JSON mode); text.format json_schema with strict: true; available “starting with GPT-4o”; refusals in a separate field
- https://platform.claude.com/docs/en/build-with-claude/structured-outputs (checked 2026-10-21; no update date): output_config.format json_schema; constrained decoding on the listed models (e.g. claude-sonnet-4-6, claude-haiku-4-5); refusal and max_tokens stops may not match the schema; enum values may differ in capitalization
- https://ai.google.dev/gemini-api/docs/pricing (updated 2026-10-19, checked 2026-10-21): Gemini 3.8 Flash input free of charge on the free tier
- The broken answer and the token odds in the picture are illustrative, labelled as such
Published Oct 21, 2026 · updated Oct 21, 2026