If you run LLMs in production, you already have a JSON repair layer — something that strips markdown fences, fixes trailing commas, and retries on schema mismatches. (If you don't, start with the production validation pattern.) Repair layers are essential. They're also where most teams make a quiet mistake: they fix the error and throw away the evidence.
Repair-and-forget is a leak
Think about what a malformed response actually is. It's not random noise — it's a measurement of how your prompt, your model, and your schema interact. When the model wraps output in markdown fences 40 times a day, that's not 40 independent accidents. It's one systematic behavior your prompt is inducing, being paid for 40 times daily: extra repair-path latency, retry tokens, and the occasional case that slips past repair into a runtime failure.
The repair layer makes each incident invisible — the response gets fixed, the request succeeds, no alert fires. Which means the pattern never gets fixed. You've built a system that pays a small tax on every request, forever, and hides the invoice.
What the failure data tells you when you keep it
Instrument your repair layer to log, for every intervention: which repair rule fired, the raw output, the prompt version, and the model version. Within a week you'll have distributions that are surprisingly actionable:
- Fence-wrapping spikes after a model update → the new version interprets your format instruction differently; one prompt line fixes thousands of repairs.
- A specific field is missing 80% of the time → your schema description buries it, or the field name is ambiguous. Rename it or add one example.
- Truncation errors cluster on one task type → those requests need a higher
max_tokensor a tighter output schema, not a retry loop. "42"instead of42from one endpoint only → the few-shot example in that prompt contains a quoted number. (Ask me how I know.)
Each of these is a one-line fix that permanently retires a whole class of repairs. But you can only see them if failures are aggregated and clustered rather than swallowed one at a time.
Close the loop: from repair log to prompt fix
The general pattern — capture failures, cluster them into patterns, fix the system, verify the fix holds — is what the agent-engineering world calls a closed feedback loop, and JSON errors are honestly the easiest place to start practicing it, because the failure signal is unambiguous: output either parsed and validated, or it didn't. No LLM-as-judge subtleties, no human labeling. The folks at ClosedLoop AI have a good breakdown of the full architecture for agent behavior in general; the JSON slice of it fits in an afternoon:
- Tag every repair event with
(rule, prompt_version, model, endpoint). - Aggregate weekly: which rules fire most, and did any rate change after a deploy or model update?
- Fix upstream for the top cluster: adjust the prompt or schema so the model stops producing that shape of error.
- Verify: replay last week's failing raw outputs against the new prompt before shipping — your repair log doubles as a regression suite for free.
- Watch the repair rate for that rule drop, then move to the next cluster.
Teams that do this consistently report the same arc: repair rates fall from "constant background noise" to "rare event worth investigating," retry costs drop, and — the part nobody expects — p95 latency improves, because the retry path was quietly dominating it.
The mindset shift
The JSON repair layer isn't just a safety net. It's a sensor that's already deployed, already positioned at the exact boundary where your model's behavior meets your system's expectations, already catching structured evidence of every mismatch. Treating its output as garbage instead of as data is the single cheapest missed opportunity in production LLM engineering.
Fix the error. Keep the evidence. Fix the cause. That's the loop.