Following up on a thread I posted yesterday about how developers detect LLM API degradation. The responses were useful enough that I want to validate a specific idea.
It is a 3 layer independent alert service:
Layer 1: Transport health alerts: Independent probes checking TTFT, error rates, and latency across major models (Claude Sonnet, GPT-4o, Gemini, Grok) every 5 minutes. Alerts you before the provider's status page updates. This part already exists and is free at tickerr - the question is whether people would pay for push alerts.
Layer 2: Capability drift alerts: A fixed canary suite that runs on a schedule and detects when a model's output behaviour has shifted, things like whether it still follows formatting instructions, whether JSON outputs are still well-formed, whether reasoning quality has changed. A drift score per model, with an alert when the score drops meaningfully from the baseline.
Layer 3: (optional add-on and phase 2): Bring your own prompts. You give us 5-10 prompts that are critical to your specific use case, we run them on a schedule and alert you if the outputs drift from your established baseline. Your prompts stay private.
Three specific questions:
- Do you think this is a useful service and would you be willing to pay for this?
- Anything else you think would make it more useful or should be included in the checks?
- What would you pay for this as a monthly service? (Ballpark is fine, even "nothing, I'd build this myself" is useful.)
If none of this is a problem you'd pay to solve, that's also fine and would save a lot of my time. 😄
[link] [comments]