What does your production LLM monitoring actually catch vs. what does it miss?

For engineers and teams shipping LLM products: curious what your real-world monitoring setup looks like vs. what it actually catches.

The common stack I see: latency tracking, token cost monitoring, basic error rates, maybe LLM-as-a-judge scoring. What I hear less about: what slips through.

A few patterns I keep running into:

- Outputs that score well on evals but fail on nuanced downstream tasks

- Failures that only surface after users stop engaging (silent failures vs. flagged ones)

- Models that improve on one metric but quietly regress on another

What's surprised you about what your monitoring missed? And has anyone actually closed the loop, where production failure data directly improved a model or routing decision?