Validation pipelines

A validation pipeline rests on one principle: the text the model produces is raw outside input until it enters your type system, exactly like whatever a user types into a form field. Pydantic states this distinction clearly: validation guarantees the output object conforms to your fields and constraints, not that the raw input had correct types to begin with.

So writing "please return valid JSON" into the prompt is not enough. That is a request, not a guarantee. The guarantee comes from the code layer that receives the output: parse it, check it against the schema, produce a typed object if it passes, reject it if it does not. Below, the model's output crosses a dashed boundary; on the left it sits in the "untrusted" zone, and only after the validator accepts it does it reach the "typed" zone on the right.

This framing can look dull, but it is the spine of the whole post. Accept the boundary, and everything else (type checks, enums, nested schemas, retries) becomes concrete gates standing on top of it.

The first confusion lives here: "the JSON parsed" and "it matches the schema" are not the same. A string can be syntactically valid JSON yet carry none of the fields, types, or values you expected. OpenAI draws this line in its own docs: the older JSON mode guarantees valid JSON, while structured outputs guarantee adherence to the schema you provide.

The SLOT paper (EMNLP 2025) makes this measurable: it counts a response as "schema correct" only if it is both valid JSON and an exact match of the target schema's key strings and value types. It even measures schema accuracy separately from content similarity, because an output can clear the syntax and still lose or distort the meaning.

Below, the same text passes through two gates. The first only asks "did it parse?" and turns green for almost anything. The second asks "do the field names and types match the schema?"; a missing field or wrong type turns it red.

The most practical way to put validation at the boundary is to write the schema as code. Pydantic does this on the Python side, Zod on the TypeScript side: you define fields, types, and constraints in code, and the library validates incoming data against that definition at runtime.

Both carry the same boundary logic. In Pydantic, model_validate_json returns a typed model instance on success and collects every error it finds into a single ValidationError on failure. In Zod, parse returns a typed deep clone on success and throws a ZodError on failure; safeParse instead returns a result object carrying either the data or the error. That is why these should be the pipeline's final parser: the place that produces either the typed value or the structured error.

Below, the same raw text enters both libraries. On the success branch it crosses the boundary and becomes a typed object; on the error branch it returns as a structured error object. The two libraries use different names, but the boundary behaviour is identical.

JSON Schema 2020-12 defines validation as assertions over a JSON instance: if every applicable location satisfies the asserted constraints, the instance is valid. That sounds abstract, but day to day it comes down to three gates. type requires the value to be the expected primitive (string, number, integer, boolean, object, array, null). enum requires the value to equal one item in a fixed allowed list. required applies only to objects and requires each listed property name to be present.

One subtlety matters: if required is omitted, it behaves like an empty array, meaning "this field is not mandatory." Your application must not assume a field is required unless the schema says so. Also, the format keyword is by default an annotation, not an assertion; if you actually want to enforce formats like email or date, you have to validate them separately at the application layer.

Below, a candidate object enters from the left and passes the three gates in sequence: type, enum, required. An object that clears all three is accepted in green on the right. On the next loop, an object that trips at the enum gate lights that exact gate red and is rejected, so which rule broke is visible.

Real outputs are not flat: an order object contains a customer, the customer contains an address, the order contains a list of line items. JSON Schema handles this with applicator keywords: properties, items, $ref apply rules to inner locations of the JSON instance. So validation does not run on one box, it walks a tree.

In Pydantic, models can contain other models and lists of models; when an error occurs the message points to a path like nested.inner, so you know at what depth it broke. Zod likewise supports nested objects, arrays, recursive objects, and discriminated unions. There is a policy choice here: what happens to unknown keys? Pydantic ignores them by default (forbid rejects, allow stores them); Zod strips them by default and throws on unknown keys in strict object mode. This "strip or reject" decision directly sets your data-loss behaviour.

Below, a nested object is validated gate by gate from the outer shell inward: order, then its customer, then that customer's address. As validation walks inward it advances in green; when the address depth produces an invalid field, that exact node lights red and the error path points to that depth.

Providers like OpenAI and Azure can now enforce schema adherence during generation (strict mode). That is powerful, but it invites a dangerous relief: "if the provider already enforces it, why should I validate?" The answer is that the provider subset is narrower than your full schema.

Azure's docs set concrete rules: in structured outputs every field must be in the required list; optionality is expressed not by a missing field but by adding null to the type (for example string or null). Objects must use additionalProperties: false; a maximum of 100 properties and five nesting levels are supported. Several type-specific JSON Schema keywords (min/max length, some pattern and array constraints) fall outside provider decoding, so your runtime validator still has to enforce them. OpenAI also notes that failure states such as refusals and incomplete output must be handled programmatically: do not blindly parse.

Below, two layers stack up. On top is the narrow zone the provider guarantees (strict schema, required-nullable, additionalProperties: false). Underneath is the gap your app-side validator closes (length, pattern, refusals, incomplete output). Without the second layer, the gaps the first layer leaves stay exposed.

If you receive the output as a stream, token by token, the object is momentarily partial: some fields have not arrived yet. Apply required as-is here and every intermediate frame is rejected as "missing field." Instructor separates this mode explicitly: it creates a partial version of a model where all fields are optional, and yields incremental snapshots until the extraction completes.

There is a cost: the Instructor docs note that validators do not run during streaming, because validation cannot be reliably applied to an incomplete state. So partial mode and required/validated mode are different operational modes. The practical pattern: while the stream fills fields you show them softly, but once the full object completes you run required and all validators one final time against the real schema.

Below, an object fills field by field via streaming. Fields that have not arrived are grey and optional; as each field arrives it validates its own mini-type and turns green. When the stream ends, a single "final required check" gate fires at the bottom and validates the whole object against the real schema one last time.

When a validator rejects, the work does not end, it begins: what will you do? Instructor gives a pipeline-shaped answer: define the model, send the request, validate the response, return the object if valid; if not, send the error context back to the LLM and retry up to max retries, then raise a ValidationError. So a validation error is the next hint handed to the model.

Guardrails AI turns these decisions into explicit policies. The Guard is the main interface wrapping the LLM call: it orchestrates validation, tracks call history, and returns the raw output, the validated output, and whether it passed, together. Validators return a PassResult or a FailResult; on failure, structured OnFail actions kick in: REASK (ask the model again), FIX (auto-correct), FILTER (drop only the failing field, keep the rest), REFRAIN (withhold the output entirely), NOOP (flag but pass), EXCEPTION (raise). That is a far richer recovery range than a single pass/fail bit, and all of it is observable.

Below, a rejected output enters a policy switch and splits into different OnFail branches: REASK returns to the model, FIX passes in corrected form, FILTER drops the failing field and passes the rest, REFRAIN halts the output entirely. Each loop highlights a different branch to show what each policy does.

No single check makes an output safe. A solid validation pipeline is layered: first the boundary itself (model output = untrusted input), then parsing (is it valid JSON), then schema correctness (type, enum, required), the depth of the nested tree, the app-side gaps the provider subset leaves open, the partial/final split for streaming, and finally the OnFail policies that fire when something is rejected.

Reading these layers together turns "do I trust the model's output?" into "at which gate, with which rule, and with which policy on failure, am I validating?" A good validation pipeline tells you not what the model produced, but where your system's boundary holds.

The model's output is untrusted input until a validator accepts it. The whole pipeline is just taking that one sentence seriously.

So before asking "do I trust the output?", ask: which gate handles parsing, which gate handles the schema, which layer closes the provider gap, and which OnFail policy fires when it breaks?