The Value Axis: Language Models Encode Whether They’re on the Right Track

Paper: 2606.17056 Authors: Nick Jiang, Isaac Kauvar, Jack Lindsey Categories: cs.CL

The Gap

Prior work on interpretability has identified directions in LLM activation space that correspond to concepts like truthfulness (Müller et al.), honesty (Li et al.), and confidence (Zhou et al.). However, these directions are typically extracted from static classifier probes on final token representations, and they fail to capture the model’s ongoing sense of whether its current strategy is likely to succeed—i.e., a dynamic, trajectory-level “value” estimate. Meanwhile, reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) alter model behaviors globally, but the internal mechanism by which the model knows “I’m on the right track” remains black-box. This paper bridges that gap by showing that such a value estimate exists as a single linear direction in early-to-mid layer activations, that it causally influences model confidence and backtracking, and that fine-tuning techniques like DPO directly modify this direction.

[Problem]          [Assumption]         [Method]               [Evidence]            [Conclusion]
+                              +                              +                              +                              +
|  LLMs lack a proxy          |  But maybe there is          |  Synthetic in-context          |  Activations along           |  Yes: a single linear
|  for "current strategy      |  a single linear             |  RL data (rollout +            |  extracted axis correlate    |  "value" axis exists.
|  will succeed" in their     |  direction encoding          |  reward) -> collect             |  with reward, verbalized     |  It encodes expected
|  internal representations.  |  expected success?           |  hidden states & apply          |  confidence, backtracking,   |  goal success and
+   (Existing probes focus    +   (Prior work found          +  linear probe.                +  code correctness.           +  causally modulates
|   on static truth/          |   linear directions for      |                                |  Steering changes            |  confidence &
|   confidence, not           |   truth/confidence.)         |                                |  behavior causally.          |  backtracking.
|   dynamic trajectory.)      |                              |                                |                              |
+                              +                              +                              +                              +

The Increment

One sentence: Before this paper, we could only measure a model’s confidence statically; after it, we can extract and manipulate a single internal variable that encodes the model’s ongoing estimate of “am I on the right track?”

Core Mechanism

The authors create a synthetic in-context reinforcement learning environment. For each trajectory, the model sees a sequence of states and actions (e.g., arithmetic problems where it must predict each step). A reward is given at the end based on whether the final answer is correct. During rollouts, they collect the hidden state activations at every token position. Then they train a simple logistic regression probe—a single weight vector—to predict the eventual reward from the activation at each step. The resulting weight vector defines the value axis: a direction in activation space that maximally correlates with the probability of eventual success.

Once the axis is identified, they can:

• Analyze activation projections: For any token, project its activation onto the value axis to get a scalar “value score.” They show this score correlates with verbalized confidence (e.g., “I’m sure” vs. “I’m uncertain”), with the presence of backtracking in rollouts, and with code correctness.
• Steer the axis: By adding or subtracting a scaled version of the axis vector during inference, they causally increase or decrease the model’s perceived value of the current trajectory. This causally suppresses self-correction (high value) or induces backtracking and exploration (low value).
• Track changes after fine-tuning: They apply DPO to make the model prefer a particular behavior (e.g., using the word “because”). After fine-tuning, the value axis shifts so that tokens where the rewarded behavior appears have higher value scores. This causes the model to act more confidently after exhibiting that behavior.

[Data Flow]
                       +---------------------+
                       | Synthetic RL        |
                       | (steps + reward)    |
                       +---------+-----------+
                                 |
                                 v
                      +----------+----------+
                      | Collect hidden      |
                      | states at each step |
                      +----------+----------+
                                 |
                                 v
                      +----------+----------+
                      | Train linear probe  |
                      | (logistic regression |
                      | from activation to   |
                      | reward)             |
                      +----------+----------+
                                 |
                    +------------+------------+
                    |                         |
                    v                         v
            +-------+-------+         +-------+-------+
            | Analysis:     |         | Steering:     |
            | project acts  |         | add/subtract  |
            | onto axis ->  |         | axis vector   |
            | value score   |         | during infer  |
            +-------+-------+         +-------+-------+
                    |                         |
                    v                         v
            +-------+-------+         +-------+-------+
            | Correlates w/ |         | Causes:       |
            | confidence,   |         | - high value  |
            | backtracking, |         |   suppresses  |
            | correctness   |         |   corrections |
            +---------------+         | - low value   |
                                      |   induces BT  |
                                      +---------------+

Let’s use a structural metaphor: think of a car’s fuel gauge.

• The car (language model) is driving through a maze (the trajectory of generation).
• The fuel gauge needle corresponds to the value axis projection (a scalar that tells the driver “how much fuel left” = how likely the current path will reach the goal).
• The synthetic RL environment is like a driving simulator where we know the exact route and final outcome (fuel or no fuel). We record the gauge reading at each moment (hidden states) and which outcomes happened. Then we learn what the gauge looks like (the axis vector) by finding the direction that best predicts whether the driver will make it.
• Once we know the gauge’s appearance, we can “tweak” it: turning the needle up (adding the axis vector) makes the driver feel confident and less likely to double-check the map; turning it down (subtracting) makes the driver nervous, more likely to backtrack and explore alternative routes.
• After a driver retraining session (DPO) where we reward using the word “because”, the gauge itself gets recalibrated so that after saying “because”, the needle jumps higher, making the driver more confident in subsequent steps.

Key Concepts

• Value Axis: A single linear direction in the model’s activation space (specifically at early-to-mid layers) whose projection predicts the likelihood that the current token-level strategy will lead to a successful final outcome. Think of it as a “progress bar” the model uses internally. Example: In a code generation task, if the model is on a correct path, the value axis projection is high; if it’s about to produce a syntax error, the projection is low.
• Activation Steering: The technique of adding or subtracting a direction vector (the value axis) from the hidden states during inference to causally influence the model’s behavior. This is like turning a dial: add → model acts more confident, less likely to correct itself; subtract → model second-guesses, backtracks more. The steering is done at a single layer (e.g., layer 14) and affects all subsequent tokens.
• In-Context Reinforcement Learning Data: A synthetic dataset where the model receives a sequence of tokens (a trajectory) and a final reward signal, without any gradient updates. The model learns from the context alone (like few-shot) but the reward is used only for the probe, not for training the model. This isolates the internal “value” representation from any external fine-tuning effects.

Framework Shift

Before (mainstream approach):        After (this paper):
+----------------------------------+ +----------------------------------+
| Static confidence probe          | | Dynamic value axis               |
|   +--- Input ---> [probe] --->   | |   +--- Input ---> [probe on      |
|   |  final token activation      | |   |  each token act] --->        |
|   |  predicts binary confidence  | |   |  score projects onto         |
|   |  (e.g., true/false)          | |   |  linear direction -> value   |
|   +------------------------------+ |   +------------------------------+
| Steering: none (only observe)    | | Steering: add/subtract axis      |
|                                   | |   causally changes behavior     |
| Fine-tuning: reward model adds   | | Fine-tuning: DPO modifies the   |
|   global policy changes, but     | |   axis direction itself, making  |
|   no internal variable tracking  | |   the model feel more confident  |
|   is available                  | |   after rewarded actions         |
+----------------------------------+ +----------------------------------+

One sentence: From static confidence classification to a dynamic, causal, and manipulable linear value axis that tracks expected goal success.

Expert Assessment

Problem choice: Real and well-motivated. The gap between static interpretability probes and the dynamic, online nature of model self-confidence is genuine. The paper picks a specific, narrow question (is there a linear trajectory-value estimate?) and answers it cleanly. However, the reliance on synthetic in-context RL data leaves open the question of whether the same axis exists in naturalistic, open-ended generation.

Method maturity: The core insight—training a linear probe on synthetic RL rollouts to discover a value direction—is elegant and not brute-force. The use of logistic regression is refreshingly simple. The authors could have tried more complex probing (e.g., nonlinear), but they show the linear probe suffices, which is a strength. The steering experiments are straightforward and convincing. A minor oversight: they don’t compare to a baseline of randomly chosen directions to show the value axis is special.

Experimental integrity: Baselines are present (e.g., comparing steering to no steering, showing proportion of backtrack tokens changes). The numbers are reported with error bars and significance tests. One concern: the synthetic RL data might introduce spurious correlations (e.g., patterns in tokens that correlate with reward) that the probe picks up instead of a genuine “value” concept. The paper partially addresses this by showing the axis transfers to out-of-distribution settings (e.g., code corruption, sensitive queries). Red flag: the activation steering might be affecting many other dimensions simultaneously; they don’t fully isolate the value axis from other potential confounds.

Writing quality: Clear and concise. The abstract effectively sells the paper. The main weakness is in the “in-the-wild” experiments section (Section 5), which feels rushed—only a few example prompts are shown without rigorous quantification. Rewriting that section with proper evaluation (e.g., controlled study on many prompts) would elevate the paper to a complete package.

Verdict: weak accept — The core finding (a linear value axis exists and is causal) is novel and reproducible, but the limited scope of in-the-wild validation and potential confounds in the synthetic data prevent a stronger recommendation.

Takeaways

Concrete things practitioners can steal:

1. Find your own value axis: If you have a language model with access to some reward signal (e.g., code passes tests, math answer is correct), collect internal activations during rollouts and train a linear probe to predict the reward. This gives you a scalar “value score” per token that you can steer.
2. Steering trick: During inference, add or subtract a scaled version of the value axis vector (e.g., +0.5 ** axis) at a specific layer. This is a lightweight way to increase model confidence (less hedging, shorter responses) or encourage exploration (more backtracking, longer chains of thought). Works even without retraining.
3. Diagnose fine-tuning effects: After DPO or RLHF, compare the value axis before and after. If a behavior you care about (e.g., using a specific phrase) gains elevated value scores, you know the model has internalized that behavior as “successful.” This gives a direct interpretability signal for what the alignment process is actually doing internally.

If you want to understand or control your model’s “feeling” about its ongoing output, the value axis is a practical, ready-to-use tool.