Most AI evaluation ends with a score. That is useful, but it does not answer a narrower question: is this particular output strong enough to release without asking a person again?
I built Chiron for the cases where the answer can be established exactly. It is an exact-or-refuse evidence gate for supported structured outputs.
For sequence-style data, Chiron recovers a constrained candidate rule and then tests it on held-out terms it did not see. Exact prediction earns verified: true; a formula that merely fits the visible data stays a candidate.
For supported claims in text, it reports each claim as VERIFIED, REFUTED, or REFUSED, then shows the coverage boundary. The surrounding free-form text is not silently approved just because one check passed.
Try a non-sensitive example here:
https://jiannotti5040.github.io/chiron/
The public evidence is deliberately bounded and runnable. The current frozen external evaluation has:
- 22 stamped outputs
- 22 externally correct
- 0 false stamps
- 12 refusals
The failure history is public too: an earlier 109-sequence sweep found 3 false stamps. Those failures were published, fixed at the root, and re-run. I think that is more useful than presenting a flawless-looking system with no stated way to be falsified.
eval/grade.py grades those frozen outputs against OEIS ground truth, and challenge.py lets you choose sequences. The protocol is here:
https://github.com/jiannotti5040/chiron/tree/main/eval
This is not a claim to certify arbitrary prose, replace experts, or solve generic "AI safety." It is meant to sit beside tracing, LLM judges, and broader evals at the point where a supported result needs an exact release condition—or an explicit stop.
The repo is source-available for noncommercial use; the full engine is commercially licensed for organizational deployment.
I would especially value counterexamples, scope objections, and examples of workflows where an exact-or-refuse gate should not be used.
[link] [comments]