When Reasoning Models Bluff: Quantifying Faithful Confidence in LLMs

Paper: 2606.03969 Authors: Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu, Arman Cohan Categories: cs.CL, cs.AI

The Gap

Existing calibration research for LLMs focuses on short-form outputs: answer a question, get a confidence score, check if it aligns with correctness. This works for Q&A models but breaks down for large reasoning models (LRMs) like o1 or DeepSeek-R1, which generate multi-step reasoning traces before answering. Users interpret these extended deliberations as signals of confidence—longer reasoning = more thoughtful = more confident. But no one has systematically checked whether the linguistic confidence in these traces (words like “definitely,” “probably,” “might”) actually reflects the model’s internal uncertainty (token probabilities, hidden states, sampling variance).

Prior work on faithful calibration (FC) assumes clean step boundaries and independent predictions. LRMs violate both: reasoning steps blur together, conditional dependencies cascade through the trace, and structural variation makes it unclear what “step confidence” even means. The field has a measurement problem disguised as a modeling problem.

Problem: LRMs generate long reasoning traces
         Users interpret length/detail as confidence signals
         |
         v
Gap:     No way to measure if expressed confidence 
         matches internal uncertainty in multi-step reasoning
         |
         v
Challenge: Prior FC metrics assume:
           - Clear step boundaries (LRMs: blurred)
           - Independent predictions (LRMs: cascading dependencies)
           - Consistent structure (LRMs: high variance)
         |
         v
Method:  New FC framework:
         - 3 internal uncertainty estimators (token/hidden/sampling)
         - Prefix-conditioned sampling to control variation
         - Linguistic decisiveness analysis
         |
         v
Evidence: Test on multiple LRMs, datasets, prompts
          Find: FC is poor across the board
                Different estimators disagree on same traces
                Reasoning ≠ better calibration
         |
         v
Conclusion: FC is a distinct alignment target for LRMs
            Current methods are fragile

The Increment

One sentence: Before this paper, we assumed reasoning models’ verbose outputs reflected genuine deliberation and appropriate confidence; after, we know their expressed confidence systematically misaligns with internal uncertainty, and we have a framework to measure this gap.

Core Mechanism

The framework has three parts: internal uncertainty estimation, linguistic confidence extraction, and alignment measurement.

For internal uncertainty, they use three independent signals. Token-level confidence aggregates next-token probabilities across the reasoning trace—like checking how surprised the model is by its own words. Hidden-state uncertainty trains probes on the model’s internal representations to predict correctness—capturing information the model “knows” but doesn’t verbalize. Sampling consistency generates multiple responses to the same prompt and measures agreement—if the model gives different answers, it’s uncertain.

The linguistic side parses reasoning traces for decisiveness markers: definitive words (“certainly,” “obviously”), hedges (“might,” “possibly”), and structural cues (conditional phrasing, self-corrections). They score each trace on a decisiveness scale from tentative to absolute.

The critical innovation is prefix-conditioned sampling. Instead of comparing traces with wildly different structures (some 50 tokens, some 500), they hold the reasoning prefix constant—generate multiple completions from the same partial trace—so structural variation doesn’t confound the uncertainty signal.

Input prompt
    |
    v
[LRM generates reasoning trace]
    |
    +---> Token probabilities --> Aggregate --> Uncertainty_token
    |
    +---> Hidden states --> Probe classifier --> Uncertainty_hidden
    |
    +---> Sample N times --> Measure variance --> Uncertainty_sampling
    |
    v
Parse linguistic markers
    |
    v
Decisiveness score (0-1)
    |
    v
Compare: Does high decisiveness align with low uncertainty?
         Does low decisiveness align with high uncertainty?
    |
    v
Faithful Calibration score

Think of it like reading someone’s poker face. The three uncertainty estimators are like watching their pupil dilation (token probabilities), heart rate (hidden states), and whether they make the same bet across multiple hands (sampling consistency). The linguistic analysis is reading their verbal tells—do they say “I’m all in” or “I might call”? Prefix-conditioned sampling is like replaying the same hand multiple times but only varying the final decision, so you can isolate confidence from luck. You’re checking: when their biology screams uncertainty, does their mouth say “maybe,” or do they still bluff with “definitely”?

Key Concepts

• Faithful Calibration (FC): Alignment between what a model internally “believes” (its uncertainty) and what it linguistically “says” (its expressed confidence). A well-calibrated model says “I’m uncertain” when its token probabilities are low or its sampled answers diverge. A poorly calibrated model says “definitely” even when internally unsure. This is not about whether the model is correct—it’s about whether its confidence expression is honest relative to its internal state. Example: A model generates “The answer is definitely 42” with 60% token probability and 50% sampling agreement. High decisiveness, high uncertainty = poor FC.

• Prefix-Conditioned Sampling: Controlling for structural variation by fixing the reasoning prefix and only varying the completion. If you ask a reasoning model the same question twice, you might get a 3-step trace vs. a 7-step trace—comparing their confidence is apples-to-oranges. Instead, generate the first K tokens once, then sample multiple completions from that shared prefix. Now all traces have the same early reasoning structure, and uncertainty variance reflects genuine confidence differences, not random structural drift. Example: Prefix = “Let’s break this down: First, we know X. Second, Y implies…” → sample 20 different endings → measure how much they agree.

• Linguistic Decisiveness: Quantifying confidence from word choice and phrasing structure. High decisiveness = definitive language (“clearly,” “obviously,” “must be”), direct assertions, no hedges. Low decisiveness = hedges (“might,” “possibly”), conditional framing (“if X then maybe Y”), self-corrections (“actually, on second thought”). The paper extracts these features using both rule-based parsing and LLM-based classification, then aggregates into a 0-1 score. Example: “The answer is 42” (high) vs. “If we assume X, then possibly 42, though it could also be 41” (low).

Framework Shift

Before (mainstream calibration):        After (this paper):

Prompt --> Model --> Answer             Prompt --> Model --> [Reasoning Trace]
           |                                       |
           v                                       +---> Token probs
      Confidence score                             |
           |                                       +---> Hidden states
           v                                       |
      Compare to correctness                       +---> Sampling variance
                                                    |
                                                    v
                                              Internal uncertainty
                                                    |
                                                    v
                                              Parse linguistic decisiveness
                                                    |
                                                    v
                                              Compare: Do they align?

Single-step, clean boundaries               Multi-step, blurred dependencies
Confidence = one number                     Confidence = linguistic patterns
Calibration = correctness match             Calibration = internal/external match

One sentence: From treating confidence as a post-hoc correctness check on final answers, to treating it as a multi-signal alignment problem between internal uncertainty and linguistic expression throughout extended reasoning.

Expert Assessment

Problem choice: Real and timely. As reasoning models proliferate, users genuinely interpret verbose outputs as confidence signals. The gap between “looks thoughtful” and “is actually confident” matters for trust. This sits at the intersection of interpretability, calibration, and human-AI interaction—a natural next step after basic LLM calibration work.

Method maturity: The three uncertainty estimators are solid but not novel individually (token probs and sampling are standard; hidden-state probes are borrowed from interpretability). The real contribution is prefix-conditioned sampling, which elegantly solves the structural variation problem. However, the linguistic decisiveness scoring still feels somewhat ad-hoc—mixing rule-based and LLM-based features without clear ablations on which matter. A simpler baseline (just count hedge words) might get 80% of the signal with 20% of the complexity.

Experimental integrity: Strong diversity in models (o1, DeepSeek, Llama) and datasets (MATH, MMLU, reasoning benchmarks). The finding that different uncertainty estimators disagree on the same traces is the most valuable result—it exposes fragility in prior work that relied on single estimators. Baselines are fair. One weakness: no user study validating that their linguistic decisiveness scores actually match human perception of confidence. They assume “definitely” signals high confidence, but in context, it might signal something else (emphasis, hedging via overstatement).

Writing quality: The abstract and introduction are excellent—clear problem statement, motivation. The methods section buries critical details (how exactly do you aggregate token probabilities over variable-length traces? What’s the probe architecture?). Table 1 should be Figure 1. The discussion overreaches slightly—claiming FC is a “distinct alignment target” is premature without showing that optimizing FC improves downstream outcomes (trust, appropriate reliance, etc.). Rewriting Section 3.2 (Internal Uncertainty Estimation) with worked examples would elevate the whole paper.

Verdict: weak accept — Addresses a real problem with a principled framework and strong empirical coverage, but the method has enough complexity and unanswered questions (Why do estimators disagree? Which linguistic features drive the signal?) that it feels more like “here’s a useful diagnostic tool” than “here’s a solved problem.” The contribution is measurement infrastructure, not a solution.

Takeaways

Prefix-conditioned sampling is the steal. Any time you’re evaluating probabilistic outputs with high structural variance (summarization, code generation, creative writing), this trick—fix the prefix, sample the suffix—lets you measure uncertainty without confounding from format differences. Implement it: generate K tokens deterministically (or fix the seed), then sample N completions from token K onward.

Multi-signal uncertainty is essential for reasoning models. Token probabilities alone are noisy for long traces. Hidden-state probes + sampling consistency give you triangulation. If all three agree, trust the signal. If they disagree, you’ve found a calibration failure mode worth investigating.

Decisiveness ≠ confidence without grounding. The paper shows that reasoning models can sound decisive while being internally uncertain. Practical implication: don’t let verbose, confident-sounding outputs bypass human oversight in high-stakes contexts. Build UIs that surface uncertainty signals (sampling variance, probability distributions) alongside the reasoning trace.

The meta-lesson: reasoning behavior doesn’t automatically solve alignment problems. Chain-of-thought makes models better at tasks, but it doesn’t make them more honest about their limitations. You need separate mechanisms to enforce faithful uncertainty communication.