Public LLM benchmarks answer a useful question: how well did this model perform on this benchmark at the time it was tested?
What they usually don't tell us is whether the same model continues behaving the same way days or weeks later.
This becomes especially interesting with API-served models, because the public model name can remain unchanged while serving infrastructure, provider configuration, routing, or potentially the underlying model implementation changes. At the same time, a score moving does not automatically mean the model changed - sampling noise, task composition, provider failures and benchmark changes can produce similar effects.
We've been looking at this as a longitudinal measurement problem rather than just a leaderboard problem.
In one historical analysis, we examined 31,352 repeated benchmark score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.
That's roughly a 3:1 difference.
Importantly, I'm not claiming this proves that providers were changing their models from day to day. There are too many possible confounders for that conclusion. What it does suggest is that temporal variation deserves more attention than it usually gets in static model comparisons.
The approach we've ended up using includes:
- repeated trials instead of single benchmark runs
- execution-based scoring where possible rather than relying on another LLM as the judge
- versioned benchmark configurations so changes to the benchmark itself don't look like model drift
- separate treatment of capability failures and provider/availability failures
- task-level analysis when an aggregate score moves
- provider/version metadata when it is exposed
- longitudinal change detection against a model's own previous measurements
- cross-model correlation to investigate whether multiple models from the same provider moved at the same time
One of the harder questions has been benchmark contamination.
Full transparency sounds ideal, but if every live task, prompt variation and hidden test is public, eventually the benchmark itself can become part of the training or optimization environment. So we've been experimenting with separating methodological transparency from publishing the entire live evaluation set.
I'm curious how people here think about this.
Do you believe LLM benchmarks should increasingly be treated as continuous monitoring systems rather than periodic rankings?
And for a live benchmark, where would you draw the line between reproducibility and keeping enough of the evaluation set private to reduce contamination?
Disclosure: I run AI Stupid Level, a commercial LLM benchmarking and drift-monitoring platform. The measurements above come from our system. I'm sharing this for discussion around longitudinal AI evaluation rather than to sell anything.
We've also published a public version of the methodology here for anyone who wants the technical details:
https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf
[link] [comments]