LLM judges are increasingly common among AI teams due their ability to automate decisions that require complex reasoning and analysis. Pairing their reasoning ability with calibrated confidence scores unlocks entirely new ways to work with AI. For one, active learning enhanced prompt optimization uses low confidence decisions to curate a golden set, allowing judges to learn human expertise with lower annotation effort. Additionally, safety classifiers for agents and chatbots can use confidence scores to reliably handle false negatives.

Uncertainty quantification for LLMs is an active research problem still in its infancy with an ongoing battle between whitebox and blackbox methods. Whitebox methods, drawing on mechanistic interpretability, read uncertainty signals from the model's residual stream, the intermediate vectors computed in each layer of the model as the weights transform your prompt into an answer. They need access to the weights, so they only work on open-source models. Blackbox methods use the tokens themselves and on occasion the token log-probabilities. Since they don't require weights most can be used on all models including closed source ones.

I compared the top 8 black box approaches with the top whitebox approach to finally put the question to rest: what LLM confidence estimation method is the best? In this post I'll explain each method in detail and how they all compare to each other.

Text Based

Verbalized confidence

If you've dabbled with confidence estimation, this was probably your first go-to. You ask the model "On a scale of 0 to 100, how sure are you?". Due to RLHF, models are trained to sound confident and agreeable which means you get less of a "You should double check my work here", and more of a "just trust me bro". One paper found that verbalized confidence scores cluster in the 80–100% band regardless of whether they're right.

Linguistic uncertainty

Humans tend to say certain words and phrases when they're uncertain. LLMs learned to speak and think from humans so maybe they do the same? (I just did it there actually) The linguistic uncertainty method counts the frequency of hedges ("maybe", "possibly", "I think") and caveats ("as far as I know", "in most cases") in the model's response.

Reasoning-length

Also grounded in human psychology this method assumes that the longer the model rambles, the less it knows. Of the text based methods this one makes the most sense given that some models are post-trained to reason longer about tasks they perceive as difficult. That said, they perform a lot better on these kinds of models (i.e. reasoning models).

Token Based

P(Answer)

Likely your second go-to after you realized the LLM already gives you probabilities for free. You read the probability of the ansswer token, and normalize it against the probabilities of the other options. In practice it can be a little tricky since the answer is rarely a single token.

P(True)

P(True) gets around the multi-token answer problem in P(Answer) by feeding the model its own answer and asking "is this correct: yes/no"? Most tokenizers treat yes and no as singular tokens making it easier to read the probability distribution.

Token based methods are the least practical in 2026 because they're incompatible with reasoning models. The thinking trajectories often mention which answer will be chosen so by the time the target token is sampled the answer is already determined: contaminating the probability distribution.

Sampling Based

Self-consistency

Sample the same question eight times at temperature 1 and count how often the model agrees with itself. It costs you 7 additional API calls and it's likely not even measuring the kind of uncertainty you want. MIT found that self-consistency mostly measures aleatoric uncertainty, the irreducible noise in the data itself. For example, a judge guessing what side a coin flip landed on, or an ambiguous task where even two experts disagree. For active learning you want epistemic uncertainty, the gaps in the model's own knowledge that more data or a better spec would close. Unfortunately for self-consistency, when the model is epistemically wrong, it just tends to be wrong again... 7 more times.

Prompt-perturbation agreement

Prompt-perturbation comes from the same lineage as self-consistency, but you nudge the framing to see if the verdict survives. In my experiments I re-ran each judgment under four reworded system prompts (be concise, be skeptical of the obvious answer, rely only on the given evidence, and drop any extra text) and scored confidence as the fraction of those four that kept the original verdict. It's a decent attempt to fix some of the issues inherent to self-consistency, but in reality it was the weakest method in the whole lineup.

Cross-model agreement

This one was the real fix: you surface the epistemic gaps by asking other models whether they agree. It is by far the strongest blackbox method and is consistent across the benchmarks, but it can be difficult to find the right set of models to use in the panel. I built the panel from three different model families so that their knowledge was complimentary rather than redundant. I also made sure all models on the panel were no more than +/- 15% accurate on the benchmarks to make sure the panel was made up of true peers and not teachers (or students). More about this later!

Mech Interp Probes (Whitebox)

For the whitebox approach we use Modaic probes via the Modaic SDK. These use ML models trained to read the LLMs internal state for signals on uncertainty and correctness. These by far have the most signal to work with. Since Modaic probes are ML models, they also have the ability to "cheat" and tune themselves to each benchmark while the other methods struggle to stay consistent across task types (binary vs multi-class, subjective vs factual, reasoning vs simple, etc) To keep the comparison fair, I show the untuned probe results alongside a probe tuned on just 100 labeled examples from the task.

Evaluation (gpt-oss-120b)

We use two metrics for evaluation, AUROC and ECE. ECE stands for Expected Calibration Error. It groups each score into bins (0-10%, 10-20%, etc) and measures the mean difference between the average confidence and the average accuracy across bins. In other words it measures how well the confidence of a prediction estimates the likelihood it is correct. The lower the ECE the better and above 0.25 is random number generator territory.

While calibration is important it is also incredibly easy to game. A particularly lazy confidence estimator can just output the accuracy of the judge itself and score a near-perfect ECE. This is why AUROC is our headline metric. AUROC is the probability that a randomly chosen correct prediction gets a higher confidence than a randomly chosen incorrect prediction. Moreover, it measures whether the estimator knows something that can discriminate good from bad. 0.5 means your estimator is no better than a coin flip 1.0 means its perfect.

I ran two judges: gpt-oss-120b, a mid-sized reasoning model, and Llama-3.1-8B, a small non-reasoning model. Each is measured on eight black-box methods (six for gpt-oss since it can't do token logprob) plus the Modaic probe in two settings, untuned and tuned on 100 examples. The tuned probes never train on examples from the held-out evaluation set. I evaluated on 1000 held-out examples for MMLU-Pro, MT-Bench, ARC-Challenge, and HaluEval Summarization. 344 for CodeJudgeBench, 300 for OR-Bench Toxic, 254 for JudgeBench, and 198 for GPQA-Diamond.

gpt-oss-120b

Benchmark	gpt-oss-120b accuracy
MMLU-Pro	79%
OR-Bench Toxic	69%
JudgeBench	82%
GPQA-Diamond	72%
MT-Bench	74%
ARC-Challenge	95%
CodeJudgeBench	83%
HaluEval Summ.	70%

AUROC (higher is better):

Method	MMLU-Pro	OR-Bench	JudgeBench	GPQA	MT-Bench	ARC	CodeJudge	HaluEval
Text based
Verbalized confidence	0.79	0.67	0.67	0.79	0.58	0.68	0.54	0.64
Linguistic uncertainty	0.79	0.74	0.74	0.82	0.60	0.71	0.67	0.60
Reasoning-length	0.69	0.82	0.44	0.75	0.60	0.67	0.58	0.60
Sampling based
Self-consistency	0.74	0.64	—	—	—	—	—	0.58
Prompt-perturbation	0.70	0.65	0.66	0.76	0.65	0.80	0.57	0.52
Cross-model agreement	0.78	0.78	0.85	0.77	0.66	0.89	0.83	0.64
Whitebox (Modaic Probe)
Modaic Probe v2 (untuned)	0.87	0.84	0.91	0.84	0.70	0.92	0.82	0.67
Modaic Probe v2 (tuned, N=100)	0.85	0.88	0.88	0.86	0.68	0.91	0.84	0.67

ECE (lower is better):

Method	MMLU-Pro	OR-Bench	JudgeBench	GPQA	MT-Bench	ARC	CodeJudge	HaluEval
Text based
Verbalized confidence	0.08	0.27	0.13	0.09	0.15	0.02	0.25	0.20
Linguistic uncertainty	0.28	0.18	0.31	0.21	0.23	0.43	0.33	0.20
Reasoning-length	0.28	0.19	0.31	0.23	0.24	0.45	0.35	0.20
Sampling based
Self-consistency	0.05	0.07	—	—	—	—	—	0.05
Prompt-perturbation	0.07	0.04	0.18	0.18	0.18	0.06	0.30	0.07
Cross-model agreement	0.10	0.15	0.07	0.13	0.17	0.04	0.09	0.21
Whitebox (Modaic Probe)
Modaic Probe v2 (untuned)	0.02	0.08	0.09	0.08	0.05	0.09	0.18	0.03
Modaic Probe v2 (tuned, N=100)	0.05	0.05	0.09	0.14	0.10	0.03	0.05	0.04

gpt-oss-120b as the judge; AUROC and ECE per benchmark. Bold is the best full-eval method per column.

riskcurve_gpt-oss

Llama-3.1-8B

Benchmark	Llama-3.1-8B accuracy
MMLU-Pro	36%
OR-Bench Toxic	61%
JudgeBench	50%
GPQA-Diamond	28%
MT-Bench	62%
ARC-Challenge	78%
CodeJudgeBench	47%
HaluEval Summ.	70%

AUROC (higher is better):

Method	MMLU-Pro	OR-Bench	JudgeBench	GPQA	MT-Bench	ARC	CodeJudge	HaluEval
Text based
Verbalized confidence	0.60	0.51	0.50	0.48	0.54	0.62	0.47	0.58
Linguistic uncertainty	0.59	0.47	0.54	0.52	0.51	0.59	0.54	0.52
Reasoning-length	0.59	0.46	0.54	0.54	0.52	0.60	0.55	0.54
Token based
P(True)	0.56	0.48	0.49	0.50	0.53	0.57	0.48	0.56
P(Answer)	0.72	0.79	—	—	—	—	—	0.66
Sampling based
Self-consistency	0.71	0.71	—	—	—	—	—	0.52
Prompt-perturbation	0.55	0.57	0.52	0.53	0.55	0.61	0.49	0.55
Cross-model agreement	0.78	0.65	0.57	0.64	0.67	0.86	0.58	0.59
Whitebox (Modaic Probe)
Modaic Probe v2 (untuned)	0.80	0.81	0.58	0.65	0.74	0.91	0.58	0.60
Modaic Probe v2 (tuned, N=100)	0.77	0.89	0.51	0.68	0.73	0.90	0.62	0.61

ECE (lower is better):

Method	MMLU-Pro	OR-Bench	JudgeBench	GPQA	MT-Bench	ARC	CodeJudge	HaluEval
Text based
Verbalized confidence	0.43	0.36	0.32	0.55	0.23	0.08	0.44	0.13
Linguistic uncertainty	0.18	0.16	0.06	0.23	0.12	0.28	0.04	0.24
Reasoning-length	0.19	0.23	0.15	0.27	0.18	0.27	0.16	0.21
Token based
P(True)	0.46	0.41	0.34	0.58	0.26	0.10	0.43	0.16
P(Answer)	0.13	0.26	—	—	—	—	—	0.03
Sampling based
Self-consistency	0.19	0.11	—	—	—	—	—	0.08
Prompt-perturbation	0.08	0.03	0.39	0.62	0.33	0.20	0.52	0.26
Cross-model agreement	0.12	0.24	0.27	0.22	0.18	0.19	0.29	0.26
Whitebox (Modaic Probe)
Modaic Probe v2 (untuned)	0.12	0.09	0.12	0.17	0.06	0.11	0.12	0.16
Modaic Probe v2 (tuned, N=100)	0.09	0.09	0.25	0.08	0.15	0.07	0.39	0.14

Llama-3.1-8B as the judge; AUROC and ECE per benchmark. Bold is the best full-eval method per column.

Findings

The more the model knows the task, the better it can estimate confidence

Look at llama's performance on MMLU-Pro vs gpt-oss's. The differentiator is accuracy. This is what makes active learning compounding. Discrimination feeds accuracy, accuracy feeds discrimination.

Prompt-perturbation underperformed self-consistency

Rewording the system prompt proves to be a weaker nudge than a temperature-1 resample. The resample actually explores the model's answer distribution, while a prompt tweak often gets shrugged off, so it flips fewer of the genuine mistakes.

P(Answer) consistently beats P(True)

I suspect this comes back to the fact that models are overconfident about their outputs. For the P(Answer) there is at least some uncertainty around which token to pick but for P(True), the log-prob seems to pick up on the model's natural aversion to saying it's wrong. Essentially, your back to the verbalized confidence "just trust me bro". Notably, P(True) is actually worse than verbalized confidence, which can at least use the model's reasoning ability to surface uncertainty.

Text-based signals work best on reasoning models

This makes sense since reasoning models are RL'd to expose their internal reasoning process "out loud", giving these methods more signal to work with.

Cross-model agreement is only as good as its panel

The signal comes from informed disagreement so choosing a competent panel is important, which is why it gets its own section below.

Multiple-choice tasks are easier

The two multiple choice tasks MMLU-Pro and GPQA consistently had high AUROC for just about every approach. My hypothesis is because they have many options its common for the judge to think two options are equally feasible. These cases are easy for most methods to pick up on as the judge talks about the tie in its reasoning and if re-sampled, will likely change its answer.

Whitebox method (Probes) win by a long shot

Unsurprising to most. Probes have a lot more signal to work with and the tuning can be a real game changer. What surprisied me the most was that many benchmarks actually didn't improve with tuning, the probe zero-shotted them outright, saturating all the observable signal.