Security and safety of structured outputs

Picture a structured output object: every field is filled, the enum values are defined, the JSON parses. Schema validation lights green. But that green light only confirms the shape is right. It says nothing about whether the inner string field carries an HTML payload, a SQL fragment, a file path, or a command.

OWASP's LLM05 Improper Output Handling points at exactly this: model output passed downstream (browser, database, filesystem, template) without validation or sanitization can become XSS, SQL injection, SSRF, path traversal, or RCE. A shape check closes none of those risks.

By sink here I mean the place where the value is actually used, where it gets interpreted and turns into an effect: a database query, an HTML page, a shell command, a file path, or a tool call. Downstream means the components that consume the model's output after it is produced. So a downstream sink is the subsystem that actually acts on the structured output's values.

Below, the same object is read twice. The validator on the left only looks at the type and lets it through; the sink on the right detonates the moment it interprets the value. The validator is necropsy, not prevention: it confirms the shape after the fact, it does not stop the intent.

OpenAI's structured outputs guide is clear about what the schema is: with strict: true it guarantees the output conforms to the supplied JSON Schema. But the same guide says separate states must be handled before parsing: a refusal (the model declining for safety) may not match the requested schema; a length limit or content filter can cut the output short.

So the schema is an interface contract: it guarantees the form of the output, not that the content is safe or authorized. Feeding a refusal, a truncated response, or filtered content blindly into a typed parser mixes a safety signal into the data. Detect which state you are in first, then parse.

Below, four output states arrive in sequence: clean parse, refusal, length truncation, content filter. Only the first flows safely into the typed parser; the other three must branch off. The schema validates the shape of all of them, but the intent of only the first.

Prompt injection (external content hijacking the model's behavior) does not only happen in the chat box. Greshake et al.'s indirect prompt injection work shows that instructions hidden in a web page, an email, or a retrieved document can steer the model without the user ever noticing. That content then leaks into a structured output field.

The result: a syntactically flawless string field can carry an operational instruction, an exfiltration payload, HTML/Markdown, a SQL fragment, or a tool argument. As the NCSC puts it, the LLM is an 'inherently confusable deputy': underneath there is no typed data-versus-command boundary, only the next token.

Below, a poisoned value leaves the model and passes the validator (green, because the type is right) and reaches one of several downstream sinks. The danger comes alive not at the validator but at the sink: HTML render, SQL builder, file path. Each sink needs its own escaping.

The fix is not to drop the shape check but to add an intent check on top of it. The first gate is schema validation: type, enum, required fields. The second gate is allow-list and policy: is this value in the permitted set, is it authorized for this origin, is this action approved? The NCSC and OWASP say the same thing: deterministic checks outside the model are essential.

The critical point: the second gate cannot live inside the LLM, because the model cannot separate typed data from a command. The authorizing mechanism must sit outside the model, tied to the origin and authority of the data. Structured output makes instructions look type-safe; that visual confidence is misleading.

Below, two objects pass through the same schema. Both have valid shapes, both clear the first gate green. The second gate (allow-list) passes one and turns the other red because its value is not on the list. Equal shape, different intent.

In tool calling the model selects a function and fills its arguments. Those arguments conform to a schema, but conforming does not make them safe. The Greshake work stresses exactly this: tool/API integration amplifies prompt injection, because the attacker can influence which API gets called and how. A path field can point under /etc, an amount can be unexpected, a url can be an exfiltration address.

The real question is not whether the argument matches the schema, but whether the selected tool and its arguments are authorized for the data origin, the user, and the requested action. OWASP recommends human approval and least privilege for high-risk actions; the NCSC says to grant privilege only to the authority of the data being processed.

Below, the model produces a tool call, the arguments conform to the schema (green). But before execution there is a deterministic policy check: if the argument is outside the permitted scope, the call is stopped before it runs. Passing the schema does not grant execution.

The risk is not only in the arguments but in which tool gets selected. The ToolHijacker work (Shi et al.) shows that a malicious tool document inserted into the tool library can steer the retrieval and selection stage. Even if the model finally emits only the selected tool name as parseable JSON, that selection came out of an already poisoned context.

So tool descriptions and metadata are executable influence over the model's routing. A structured tool-choice output does not guarantee the chosen tool is safe, authorized, or correctly scoped. The work tests defenses like StruQ, SecAlign, and perplexity detectors and finds each insufficient on its own.

Below is a tool registry. A poisoned tool document (red) inflates its selection score and pulls the choice to itself; the structured output is still valid JSON. The right defense: treat the registry as a supply-chain asset, verify provenance, sign and pin it, and keep untrusted extensions separate.

So far the value was data (the data plane). But the schema itself can be an attack surface. Zhang et al. describe a structured-output-specific control plane risk: the grammar/JSON Schema used for constrained decoding shapes generation. If the malicious intent is placed in the schema rather than the visible prompt, classic input/output defenses miss it.

Their EnumAttack hides malicious content in enum values; their DictAttack splits the payload between prompt and grammar. The reported attack success rates are unsettlingly high. The takeaway is clear: a schema supplied by a user, tenant, plugin, or data pipeline is not trustworthy.

Below, an enum field carries a poisoned value among innocent-looking options. Because the decoder must conform to the schema, it can emit that value; a data-plane defense only looks at the data and misses the control plane. Treat the schema as trusted code/config, not user data.

The consumer of the JSON is the real attack surface. Ajv's security notes are explicit: JSON Schema can replace some sanitization but not other API security considerations. And untrusted schemas are as dangerous as application code: a schema that is untrusted or generated from untrusted data can cause stack overflows, slow compilation, and slow validation.

Even trusted schemas have DoS pitfalls: regex-heavy pattern/format can cause ReDoS, allErrors: true is discouraged in production because it does not stop at the first error on hostile input, and browser validators that are not precompiled may require unsafe-eval and break the CSP/XSS balance. So the validator's configuration and the schema's provenance are part of the security model.

Below, each downstream sink shows its own defense: encoding for HTML, parameterization for SQL, canonicalization for file paths, policy plus human approval for tools. Not one generic filter, but a sink-specific control. The structured output stays untrusted until it gets here.

Structured output largely solves shape failures; but a valid shell can carry an invalid intent.

So before saying "the schema passed", ask: which sink does this value go to, under whose authority, and which deterministic check will stop it? Validate the type, never trust the value.