Structured output reliability across models

You write a single JSON Schema: a few required fields, one optional field, an enum, some nesting. Then you send it to three providers. The results often differ. On one, the output conforms fully (pass). On one, the JSON is valid but part of the schema has silently vanished, an unsupported constraint dropped (partial). On one, the model just misses the constraint (fail).

The reason is that "structured output" is not one standard. Each provider supports a different subset of JSON Schema, enforces it with a different engine, and handles unsupported features differently. JSONSchemaBench makes the picture concrete: the best framework supports roughly twice as many real-world schemas as the worst.

Below, one schema fans out to three providers and each lands in a different outcome cell. Same input, different output. The rest of the post explains where that difference comes from.

When a provider says "we guarantee a schema-compliant response", it is easy to conflate two very different things. The first is syntactic correctness: is the output valid JSON, are the required fields present, are enum values from the allowed set, are the types right? The second is semantic correctness: are the values inside the fields actually right, sensible, and what the user wanted?

Constrained decoding guarantees only the first. OpenAI and Anthropic say structured output produces a schema-valid response, but that does not mean the content is correct. Google's Gemini docs warn this explicitly: even syntactically correct JSON still needs application-side value validation and robust handling for schema-compliant but semantically wrong values.

So the "guarantee" is a guarantee of shape, not content. Below are two gates: structured output always clears the first (valid shape), but it does not clear the second (correct value), which you have to validate yourself.

Under most native modes and open-source engines sits the same idea: constrained decoding. At each step the model produces probabilities for the next token in its vocabulary. Constrained decoding masks every token that would make the current partial output schema-invalid, zeroing its probability. The model can only sample a token that keeps the schema valid.

OpenAI's account is telling: model training alone was not enough (in their own evals training plateaued at 93%), so they paired training with deterministic constrained decoding. The same approach is formulated model-agnostically in Willard and Louf's Outlines work: the method needs no retraining and can sit between many models and the systems that consume them.

Below, two generation lanes run side by side. The top, unconstrained lane sometimes emits an out-of-schema token (red) and the output breaks. The bottom, constrained lane masks that same invalid token (struck through), so it never emits an invalid token at all.

All three big providers now offer a JSON Schema interface, but each with a different operational contract. OpenAI exposes two surfaces: strict tool/function calling, and response_format with json_schema for the final answer. It requires strict: true, all fields must be required (optionality is emulated with a union type including null), and objects must use additionalProperties: false.

Anthropic separates "JSON outputs" for the response shape from "strict tool use" for validated tool names and inputs; the two can be used independently or together. It says structured outputs are schema-compliant via constrained decoding, but its SDKs may transform unsupported constraints: removing minimum, maximum, minLength and folding them into descriptions, then validating the response client-side against the original schema. That weakens any naive portability assumption.

Google positions Gemini structured output for a predictable response shape and function calling for taking actions mid-conversation. Gemini also supports a subset of JSON Schema, and the docs still recommend you validate values. All three converge on JSON Schema, none support all of it, and their transformation behaviors differ.

Here is the real portability problem: "JSON Schema support" is not uniform. OpenAI supports thousands of object properties and several levels of nesting, but requires all fields to be required and objects to be closed (additionalProperties: false). Anthropic SDKs may silently strip numeric and length constraints. Gemini accepts only selected object, string, number, and array properties.

The consequence: a schema that works flawlessly on one provider is either rejected or silently pruned on another. Silent pruning is the sneakier one, because the output looks valid but the constraint you expected (say minimum: 0) was never applied. You won't notice without testing.

The practical way out is to design the schema around the lowest common denominator: required fields, explicitly closed objects, shallow structure, simple types and enums, and to check numeric/semantic constraints outside, in your own validation layer. Below, a rich schema passes through the providers' subsets; a different feature drops on each, and the core that survives is the part that is portable everywhere.

Going off-provider and running your own constrained decoding does not end the problem either. Outlines, XGrammar, llama.cpp and similar engines promise the same guarantee but differ in grammar class, tokenizer handling, schema coverage, compilation latency, and per-token cost. Outlines formulates guided generation as state transitions over a finite-state machine, with vocabulary indexing giving average O(1) masking for regular expressions.

Koo, Liu and He's automata-based work sharpens the underlying difficulty: subword tokenizers are ambiguous and misaligned with formal grammar tokens, so naive alignment either harms quality or creates many special cases. They model it cleanly with automata and finite-state transducers, reporting constraint compilation about 7000x faster and provably correct.

XGrammar emphasizes another axis: cost. Flexible CFG-based structured generation can be expensive; XGrammar splits vocabulary tokens into ones that can be prechecked (context-independent) and ones needing a runtime stack check (context-dependent), reporting up to 100x speedup over existing solutions. So "schema support" is not a commodity: engine design sets both reliability and latency.

Structured output is not a portable contract; it is provider-dependent behavior. What is guaranteed is shape, not content, and the cost of shape differs in every engine.

What grounds these differences is JSONSchemaBench. The work pairs over 10,000 real-world JSON schemas with the official JSON Schema Test Suite, a far harder test bed than hand-picked toy schemas. It compares Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini on three axes: efficiency, JSON Schema feature coverage, and output quality.

The three axes must be read separately, because they measure different things. Schema compliance (does the output hold the schema), schema feature coverage (how many real-world schemas the framework supports), and task quality (is it semantically correct) are independent signals. A framework can show high compliance but low coverage, supporting few schemas while holding the ones it does.

The headline findings: constrained decoding can speed generation by about 50% in some settings; real-schema support varies sharply across frameworks (the best roughly twice the worst); and downstream task accuracy improves by up to 4% on the reported tasks. Below, coverage bars fill per method, with the three evaluation axes in separate boxes on top.

To pull it together: structured output is not a portable standard across providers. The same schema conforms fully in one place, is silently pruned in another, and rejected in a third. The "guarantee" is a guarantee of shape; you still validate semantic correctness. Native modes and open-source engines share the same constrained decoding idea but come with different schema subsets, different transformations, and different costs.

The practical rule is simple: design the schema around the lowest common denominator, put numeric and semantic constraints in your own validation layer, and most importantly, run a real test with your real schema on every target provider. Portability is not free, but if you measure which feature drops where, you can choose the part that travels.

A schema looks like a contract, but it gets renegotiated at the provider boundary.

So before saying "the schema is guaranteed", ask: on which provider, with which subset, under which transform, and at what cost is it guaranteed?