Legibility Is Not Interpretability: CoT Step Importance Is Only Partly in the Text

Paper: 2609.04194 Authors: Kevin Du, Alexander Hoyle, Laura Ruis, Acyr Locatelli Categories: cs.CL, cs.LG

The Gap

Chain-of-thought traces look like a window into how a model reached its answer, and a growing body of practice treats them as one: LLM judges diagnose errors from the trace, evaluate faithfulness from it, and provide step-level supervision through process reward models and generative critics.

All of that rests on an assumption that is easy to state and had not been tested: the text of a reasoning step carries information about its functional role. A readable step and an important step are not the same thing, and the difference matters precisely for the applications that read traces. A process reward model trained on what looks like the key step is rewarding the legible step, which may not be the load-bearing one.

The distinction is between legibility — the trace is readable — and interpretability — the trace tells you what the model actually did.

   CoT TRACES TREATED AS A WINDOW INTO DECISION-MAKING

   LLM judges DIAGNOSE ERRORS from the trace
   faithfulness is EVALUATED from the trace
   step-level supervision (process reward models,
     generative critics) is DERIVED from the trace
        |
        v
   ALL REST ON AN ASSUMPTION, EASY TO STATE AND UNTESTED:
     the TEXT of a reasoning step carries information about
     its FUNCTIONAL ROLE
        |
        v
   [WHY THE DISTINCTION MATTERS]
     a READABLE step and an IMPORTANT step are not the same
     -> a process reward model trained on what LOOKS like the
        key step is rewarding the LEGIBLE step, which may not
        be the LOAD-BEARING one
        |
        v
   LEGIBILITY        = the trace is readable
   INTERPRETABILITY  = the trace tells you what the model
                       actually did

The Increment

One sentence: Before this paper, the informational content of a reasoning step was assumed; after it, step importance defined as expected-reward advantage and measured by Monte Carlo rollout is only partially recoverable from the trace’s text, which is a caution for process reward modeling and CoT faithfulness work.

Core Mechanism

The contribution is the operationalisation, because it converts a philosophical worry into a measurable question. A step’s importance is its advantage: the change in expected reward — such as producing the correct final answer — from including that step, estimated by Monte Carlo rollouts. That is a functional definition: a step matters if its inclusion changes the outcome distribution, regardless of how it reads.

Establishing that ground truth is what makes the rest measurable. Then the question becomes empirical: can judges identify high-advantage steps?

The answer is a bounded one, and its shape is the finding:

Sufficiently capable LLMs can outperform a prevalence baseline. So there is some signal in the text about functional role — a judge doing better than the base rate of important steps is evidence that readability correlates with importance to a degree.

But they fall well short of a noise ceiling. The ceiling is the important reference point. Falling short of it means the text does not contain enough information to identify important steps reliably, even in principle for this method — so the gap is not a shortfall of effort or scale but of available signal.

Fine-tuning a model as a step-level critic yields strong improvement for incorrect responses but remains distant from ceiling for correct responses. This asymmetry is the sharpest part of the result. When the response is wrong, step importance is much more recoverable — plausibly because a wrong answer often has an identifiable point where it went wrong, which the text reflects. When the response is correct, importance stays largely hidden. So the regime that matters most for process reward modeling — supervising the steps of successful reasoning — is precisely the regime where the text is least informative.

The paper’s own framing of the implication is careful: the findings contribute to a growing body of chain-of-thought faithfulness work that cautions against treating the legibility of reasoning traces as interpretability, and they carry particular implications for process reward modeling. That last clause is the actionable one, and it follows directly: if a critic cannot reliably identify which steps of a correct trace matter, then a reward model built on critic judgements is supervising against a partially unobservable target.

   OPERATIONALISATION: WHAT MAKES A STEP IMPORTANT

   a step's ADVANTAGE = the CHANGE IN EXPECTED REWARD
     (e.g., producing the correct final answer)
     from INCLUDING that step
        <- estimated via MONTE CARLO ROLLOUTS
        <- a FUNCTIONAL definition: a step matters if its
           inclusion changes the outcome distribution,
           REGARDLESS OF HOW IT READS

   THEN THE EMPIRICAL QUESTION: can judges identify
   HIGH-ADVANTAGE steps?
        |
        +-- SUFFICIENTLY CAPABLE LLMs beat a
        |   PREVALENCE baseline
        |     -> there IS some signal in the text about
        |        functional role
        |
        +-- BUT fall WELL SHORT OF A NOISE CEILING
        |     -> the text does not contain enough information
        |        to identify important steps reliably
        |     -> a shortfall of AVAILABLE SIGNAL, not of
        |        effort or scale
        |
        +-- FINE-TUNED STEP-LEVEL CRITIC
              STRONG improvement for INCORRECT responses
              remains DISTANT FROM CEILING for CORRECT responses
              <- the sharpest finding: the regime that matters
                 most for process reward modeling -- supervising
                 the steps of SUCCESSFUL reasoning -- is exactly
                 where the text is least informative
                 (a wrong answer often has an identifiable
                  point where it went wrong; a right one may not)

Think of it as reading a chess player’s narrated thoughts and trying to work out which move actually decided the game. The narration is fluent and often plausible, and some of what it says is genuinely load-bearing. But a move that sounds like a dramatic turning point may have been forced, while a quiet move the player barely mentions may have been the one that mattered. If you trained a coach on which moves sound important, you would be teaching them to praise the narratively satisfying ones. And notice where the method would work best: on games that were lost, the mistake is often visible in the narrative because it produced a clearly bad position. On games that were won, the decisive move can be invisible — and that is exactly the case where you want the coaching.

Key Concepts

• Advantage as step importance: the change in expected reward from including a step, estimated by Monte Carlo rollout. Defining importance functionally — by effect rather than by appearance — is what makes the question measurable and separates it from readability.
• Legibility versus interpretability: a readable trace versus a trace that identifies what mattered. Conflating them is the error the paper targets, and it is specifically the error that process reward models would inherit.
• The noise ceiling as the reference: beating a prevalence baseline shows signal exists; falling short of the ceiling shows it is insufficient. Both halves are needed to state the result accurately.
• The correct/incorrect asymmetry: step importance is much more recoverable for wrong answers than right ones. It matters because supervising successful reasoning is the main use case, and that is the harder regime.
• Partial recoverability as the claim: not that traces are uninformative, but that functional role is only partly in the text. Keeping the claim at that strength is what makes it credible and still consequential.

Framework Shift

Before (trace as a faithful account):
  read the reasoning trace, diagnose errors from it,
  evaluate faithfulness from it, derive step supervision
  -> assume the text of a step carries its functional role
  -> reward the steps that LOOK like the key ones

After (importance measured functionally):
  advantage = change in expected reward from including
              the step, via Monte Carlo rollout
  -> capable judges beat a prevalence baseline but fall
     well short of a noise ceiling
  -> a fine-tuned critic is strong on incorrect responses,
     distant from ceiling on correct ones
  -> step importance is only PARTLY recoverable from text

From treating a legible trace as an interpretable one, to measuring a step’s importance by its effect on the outcome and finding the text carries only part of it, the core shift is that reading a model’s reasoning is not the same as knowing which of its reasoning mattered.

Expert Assessment

Problem choice: Excellent, and it is aimed precisely at the practice most affected. Chain-of-thought faithfulness is discussed at length, but process reward modeling consumes step-level judgements at scale — so a finding that step importance is partially unobservable from text bears directly on a deployed technique rather than on an interpretive debate.

Method maturity: Defining importance as advantage and estimating it by rollout is the right operationalisation: it is model-internal, causal, and independent of how the step reads, which is exactly what a test of the legibility assumption requires. Establishing a noise ceiling as a reference is a methodological strength — it makes “judges are imperfect” into a statement about the available signal. The fine-tuned critic probes how far the recoverable part can be pushed, which is the appropriate follow-up to the judge measurement.

Experimental integrity: The asymmetry between correct and incorrect responses is reported rather than smoothed, and it is the result least favourable to the applications the paper discusses. The claim is kept at the strength the evidence supports — partial recoverability, not uninformativeness. The open question is scope: the Monte Carlo advantage estimates depend on the task’s reward being well defined and on enough rollouts to estimate the difference, and how the findings transfer to tasks with sparser or noisier reward, such as open-ended generation, is not established.

Writing quality: The advantage definition is stated exactly once and used consistently, which is what makes a functional definition usable. Because the actionable consequence is a warning about process reward models, a short passage on what a practitioner should do — for instance, evaluating a critic against rollout-based advantages before trusting its supervision — would convert the caution into a procedure.

Verdict: strong accept — it tests the assumption that makes step-level supervision possible, operationalises step importance causally, and reports that the text carries only part of it, with the sharpest gap exactly where the application needs it most.

Takeaways

• Measure importance by effect, not by appearance. A step’s advantage is the change in expected reward from including it, and that is not the same as how much it sounds like it mattered.
• Establish a ceiling before judging a gap. Beating a baseline and falling short of a ceiling are both necessary halves of a claim about available signal.
• Check which regime your supervision sits in. Step importance is more recoverable for wrong answers than right ones, so supervising successful reasoning is the harder case.
• Validate a step-level critic against rollout-based advantages. If importance is only partly in the text, critic judgements are supervising against a partially unobservable target.