Repair loops and fallback chains

A repair loop rests on one idea: do not accept output blindly, validate it; if it fails, hand the error back to the model and ask again. Microsoft TypeChat frames this as the library's contract: construct the prompt from types, validate that the model response conforms to the schema, and repair non-conforming output through further model interaction. Repair is not a separate manual chore, it is part of the flow.

The loop has four stops: generate (produce), validate (parse and schema check), feedback (collect what broke), repair (ask again with the error attached). If validate passes the loop closes and the output is accepted; if not, the error is fed into the next generation's input. This is exactly the generate → feedback → refine loop that the Self-Refine paper formalizes.

Below, a broken JSON enters the loop: the validator rejects it, the error flows back to the model as feedback, the model emits a corrected version, and this time the validator passes. The flow runs continuously and softly shows where each turn is.

Before reaching for repair there is a layer that minimizes errors at the source: constrained decoding, forcing generation to follow the schema directly. OpenAI's Structured Outputs says it guarantees adherence to a supplied JSON Schema, reducing the need to validate or retry for incorrect formatting. LangChain draws the line between ProviderStrategy for models with native support and a tool-based ToolStrategy where there is none. So guarantee the format up front when you can.

But "valid JSON" and "schema-valid JSON" are not the same. JSON mode produces valid JSON without guaranteeing that fields match the types and rules; validation libraries and retries may still be needed. And even when constrained decoding fixes the format, it cannot fix meaning: a date can be a valid string yet nonsensical, an id can look valid yet have no row in the database. These are semantic errors, and the repair loop is exactly where they belong.

Below are two gates: structural (format) and semantic (meaning). Constrained decoding largely closes the left gate; outputs that cannot clear the right one fall into the repair loop.

The heart of the repair loop is the error you hand back to the model. Many systems treat the validation error not as a log line but as the core repair signal. In Guardrails AI, OnFailAction.REASK asks the model for output that satisfies the validator and adds, to the reask prompt, information about which quality criterion failed and why. In LangChain, ToolStrategy.handle_errors can retry with a default message, a custom message, or selected exception types. Error feedback is an implementation surface, not a prompting trick.

The feedback should be specific and actionable. Self-Refine, CRITIC, and Instructor point the same way: "try again" is weak; "field price must be a number but a string arrived" is strong. CRITIC further warns that pure self-correction without an external tool can yield limited gains or even degrade. So the feedback is usually a concrete error from a parser, a JSON Schema validator, or a Pydantic model, not the model's own guess.

Below, the same broken output is given two kinds of feedback. The left prompt is vague ("invalid, fix it") and the loop spins; the right prompt names exactly which field broke which rule and corrects in a single turn.

The obvious danger of a repair loop is this: if the model cannot fix it, the loop spins forever, burns tokens, and inflates latency. So retries must always be bounded. Instructor makes this concrete: you initialize the client with max_retries, and the Pydantic validation error feeds back to the model. In the docs example, max_retries=2 means the initial attempt plus two retries, so three attempts in total.

The bound is not just a number, it is a policy. What happens when the budget runs out must be defined in advance: propagate the failure, raise an exception, or move to a fallback. Silently returning the last (probably still broken) output is the worst choice, because it hides the failure. Self-Refine and CRITIC make the same structural point: the loop must stop at a fixed iteration count or an explicit stopping condition.

Below, a retry budget counter decrements on each failed attempt. Three scenarios sit side by side: an early success preserves budget; a late success still lands before exhaustion; and if the budget hits zero, the loop stops and hands off to a fallback.

Not every error needs a round trip to the model. For narrow, mechanical format errors, deterministic repair can be cheaper and safer: code fixes the output without waiting for the model. Guardrails separates this as the FIX action, repairing the output programmatically. FILTER strips invalid fields, and FIX_REASK first tries the deterministic fix and reasks only if validation still fails.

The split must be clear. A missing closing brace, quotes wrapping a number, an extra comma: when fixing these in code is safe, it beats asking the model. But semantic violations like "does this id exist in the database" or "does this amount comply with policy" cannot be repaired deterministically; they call for reask, retry, or an application/human fallback. Applying a deterministic fix in the wrong place silently produces wrong data, which is more dangerous than broken JSON.

Below, an output passes a decision point. If the error is mechanical/format, it takes the left path and is fixed in code; if it is semantic, it takes the right path back to the model as a reask. The two branches are distinct colors and both are legible.

A single strategy is not always enough. A fallback chain is a cascade where, when one stage fails, control passes to the next. The typical order is: provider-native constrained decoding first; where that is missing or insufficient, validator-guided reask; then partial salvage via deterministic fix or filter; and if all fail, fail closed, raising an exception or escalating to human review. Guardrails' action taxonomy (retry, reask, fix, filter, refrain, exception) supplies the chain's building blocks.

The chain's strength is that each rung tries a different lever from the one before it: change the output mode, change the repair strategy, try a more constrained prompt, or harden the failure policy. LangChain's provider-native → tool-strategy split is one example. What matters: each rung is bounded, each handoff is deliberate, and the end of the chain is always a defined terminal state (accept, reject, or human).

Below, control slides down the chain. Each stage is tried in turn; a failed one flashes red and hands off to the next, a passing one flashes green and stops the chain. There is always a terminal state at the bottom.

Retries do not have to be stateless. The Reflexion paper turns binary or scalar feedback from the environment into a verbal summary stored in memory, so later attempts can avoid earlier mistakes. In a structured-output context this means each failed attempt leaves a short failure summary: which field, which rule, which value broke.

This is especially valuable in a fallback chain. A plain retry on the same call may not need memory; but when the chain switches model, prompt, parser, or tool strategy, a compact summary of prior failures makes the next rung far more accurate. CRITIC points the same way: carrying critiques from external validation beats retrying blind. So do not restart retries from scratch; carry what broke to the next attempt.

Below, three attempts fail in turn but each leaves a note in memory. As the memory on the right fills, the next attempt takes those notes as input; eventually the accumulated context brings success.

No single mechanism recovers a broken output on its own. A robust design is layered: constrained decoding to guarantee the format at the bottom; a repair loop above it driven by validation and specific error feedback; a retry budget framing it; deterministic fix where appropriate; and at the end of the chain, always a defined terminal state (accept, partial salvage, or human).

Reading these layers together turns "why did the model emit broken output?" into "when broken output arrives, what does the system do, in how many turns, at what cost, and where does it stop?" A good recovery design does not expect the model to be perfect; it anticipates broken output, bounds it, and always binds it to a terminal state.

Robust structured output is not a system where the model never errs; it is one that, when the model errs, recovers in a bounded, observable, and terminal way.

So before saying "the model returned broken output", ask: at which layer do I catch it, with how many retries do I bound it, how specific is my error feedback, and where does the chain stop?