When are likely answers right? On Sequence Probability and Correctness in LLMs

Paper: 2606.27359 Authors: Johannes Zenn, Jonas Geiping Categories: stat.ML, cs.LG

The Gap

Existing decoding methods (greedy, beam search, top-k, nucleus sampling, etc.) all push probability mass toward more likely sequences — either locally (token-level) or globally (sequence-level). The implicit assumption is that higher model probability correlates with correctness. However, no prior work had systematically quantified this relationship across multiple dimensions: methods, hyperparameters, prompt-answer pairs, and repeated responses. Prior evaluations either assumed the correlation (e.g., in MBR decoding or self-consistency) or only tested isolated cases. This paper fills the gap by asking: when does this assumption hold, and when does it break?

[Problem: Decoding methods shift prob to likely outputs]
        |
        v
[Assumption: higher seq prob -> more correct answers]
        |
        v
[Method: Quantify prob-correctness alignment at 4 levels]
        |
        v
[Evidence: alignment is conditional; breaks across methods/hyperparams/repeats]
        |
        v
[Conclusion: decoding cannot be expected to reliably improve correctness]

The Increment

One sentence: Before this paper, it was common to treat sequence probability as a proxy for correctness; after this paper, we know that probability is only a correct proxy when comparing different answers to the same prompt across a dataset, and fails entirely for choosing among decoding methods or hyperparameters, or for picking among repeated responses to the same prompt.

Core Mechanism

The authors propose no new algorithm. Instead, they design a controlled measurement framework. They take a fixed set of prompts (from benchmarks). For each prompt, they generate continuations using different decoding methods (greedy, beam, top-k, top-p, typical, etc.) and different hyperparameter settings (temperature, top-k range, beam width, etc.). For each generated continuation, they record both its sequence probability (the log-prob of the entire output under the model) and whether it is correct (based on benchmark labels). They then compute correlation measures and compare across four levels:

• Across methods: does a method that yields higher probability sequences also yield higher accuracy?
• Across hyperparameters within a method: does increasing temperature (which usually lowers probability) hurt accuracy? Or does narrowing top-k (which increases probability) help?
• Across prompt-answer pairs within a fixed dataset: among all answers to different prompts, are those with higher probability more likely to be correct?
• Across repeated responses to the same prompt: if you sample multiple times from the same model with the same hyperparameters, does the most probable response tend to be the correct one?

The data flow: Prompt -> Model (with fixed decoding config) -> Sequence (utterance) -> Compute log-probability -> Compare to ground-truth correctness label -> Aggregate into per-level correlation stats.

  +---

![Concept animation](/arxiv-visuals/when-are-likely-answers-right-on/ConceptScene.gif)

------+   +-----------+   +--------+   +---------+
  | Prompt  |-> | LLM Model |-> | Output |-> | Prob    |
  +---------+   +-----------+   +--------+   +---------+
                                            \|/
                +-----------------+    +-------------+
                | Correctness?    |<- | Label from  |
                | (binary)        |    | benchmark   |
                +-----------------+    +-------------+
                     |
                     v
     +-------------------------------+ 
     | Compare prob vs correctness   |
     | at four levels:               |
     | 1) across methods             |
     | 2) across hyperparameters     |
     | 3) across prompt-answer pairs |
     | 4) across repeated responses  |
     +-------------------------------+

Structural metaphor: Think of sequence probability as a thermometer reading, and correctness as whether it’s actually raining outside. You have four scenarios:

• Across cities (different prompts): temperature readings correlate with rain probability — warmer cities tend to be rainier? That’s the “across prompt-answer pairs” level; it holds.
• Across weather stations in the same city (different responses to the same prompt): temperature readings from different stations are all similar and don’t tell you which one is truly raining—that’s the “repeated responses” level; probability doesn’t discriminate because all samples have nearly equal probability.
• Changing the thermometer type (different decoding methods): a mercury thermometer and a digital thermometer give different readings; the higher reading doesn’t tell you more rain—that’s “across methods”; the alignment breaks.
• Adjusting the measurement unit (hyperparameters like temperature): moving from Celsius to Kelvin shifts all numbers; a higher number doesn’t mean more rain—that’s “across hyperparameters”; also broken.

So the thermometer (probability) is only useful when comparing apples to apples across different cities; otherwise it’s a distraction.

Key Concepts

• Sequence probability: The conditional probability of an entire generated token sequence given the prompt, computed as the product of token-level probabilities (in log space). It’s a single scalar that the model itself assigns to its output. High sequence probability doesn’t mean “the model is confident” in any human sense — it means the output lies in a high-density region of the model’s distribution. The paper uses it as the independent variable.

• Correctness: A binary label from a benchmark dataset (e.g., whether the answer matches the expected output on GSM8K or MMLU). Important nuance: correctness is defined externally, not by the model. The entire paper hinges on comparing an intrinsic property (probability) with an extrinsic property (correctness).

• Decoding methods: Algorithms that convert the model’s token probability distribution into a concrete output sequence. Greedy always takes the most likely token. Beam search keeps a set of high-probability partial sequences. Top-k and top-p restrict the sampling pool. Each method implicitly biases toward higher or lower sequence probabilities. The paper treats the method itself as a treatment variable.

Framework Shift

Before (mainstream assumption):        After (this paper):

  [model distribution]                    [model distribution]
        |                                        |
  [decode method]                          [decode method]
        |                                        |
  [high prob output]        +---X---?---X-> [output correctness]
        |                    |            |
  [more correct?]            +--> prob helps across pairs
                                     but not across methods
                                     , hyperparams, or repeats

From “higher sequence probability → more correct answers” to “sequence probability predicts correctness only when comparing different answers to different prompts under the same decoding conditions.” The core shift is context dependence: the signal in probability is dataset-relative, not absolute or invariant.

Expert Assessment

Problem choice: Real gap. Prior work often assumed the link between likelihood and correctness (e.g., in self-consistency and MBR). Showing that the link is conditional is both practically important and theoretically underexplored. The paper sits at a critical junction between decoding theory and applied LLM usage.

Method maturity: Clever in its simplicity — no new algorithm, just careful measurement. However, the analysis is entirely empirical. There is no theoretical explanation for why the breakdown occurs (e.g., why across methods the correlation disappears). A more mature paper might attempt a causal model or prove a bound. The approach is brute-force experimental, which is fine for a gap-filling paper, but lacks depth.

Experimental integrity: Baselines are fair (they compare standard decoding methods). The four-level decomposition is clean. However, I worry about statistical power: the number of repeats per prompt (e.g., 10 or 100) could affect the across-repeated-responses conclusion. The paper does not report confidence intervals or effect sizes thoroughly. Another red flag: all experiments use a single model family (likely Llama or similar)? The paper abstract mentions “models” plural, but the actual experiments may be limited. If they only test one base model, generalization claims are weak.

Writing quality: Well structured, but the results section is dense with numbers. The authors could cut the steady stream of correlation values and instead show a summarizing decision tree or a cheat-sheet table. The weakest section is the conclusion — they list practical guidance but don’t connect it back to the four-level framework cleanly. Rewriting the conclusion as a “when to trust probability” rule set would elevate the paper.

Verdict: weak accept — the question is important and the evidence is sufficient to shift practice, but the paper lacks theoretical grounding and statistical rigor.

Takeaways

• Don’t tune hyperparameters to increase sequence probability: higher temperature (lower probability) sometimes improves accuracy; lowering temperature (higher probability) can hurt. The paper’s data shows that accuracy and probability are not monotonic across hyperparameters.
• Self-consistency (majority vote across samples) relies on the assumption that the most frequent answer is correct — but note that frequency correlates weakly with average probability. This paper suggests self-consistency might work not because the most probable answer is right, but because majority voting averages out random noise. The “correct answer” may not be the one with highest sequence probability.
• If you must use probability for selection (e.g., in MBR), restrict comparisons to outputs generated under the same decoding method and same hyperparameter configuration. Cross-method probability comparisons are meaningless.
• For verifier-free self-improvement (e.g., training on high-probability outputs), beware that high probability can be a misleading target. Better to filter by correctness (if labels exist) or by consistency across multiple samples.