Been shipping structured JSON output from LLMs in production for a health app. Here's what I've learned about reliability.
The problem: get a 70B model to return valid JSON matching a strict schema, every time.
What I tried:
Attempt 1: "Return JSON." No schema. 40% valid output.
Attempt 2: Detailed schema in prompt. 75% valid.
Attempt 3: JSON mode enabled (Groq/OpenAI/Anthropic all support). 92%.
Attempt 4: JSON mode + schema validator + retry loop with error surfaced back. 99.5%.
What still fails:
- Emoji in fields (invalidates JSON parsing)
- Very long generated fields (context length errors)
- Rare "the model just doesn't return JSON" (0.5% baseline you can't kill)
For production, my flow:
LLM call in JSON mode with schema
Parse. If fails, log the raw output for analysis
Validate against Zod schema
If schema fails, retry ONCE with the validation error in the prompt
If still fails, use a static fallback
Model tier matters less than I expected. Prompt scaffolding matters more.
Question: anyone doing something more sophisticated? Curious about output-guided generation via Outlines or LMQL in production.
[link] [comments]