Emergent Search and Backtracking in Latent Reasoning Models

Paper: 2602.08100 Authors: Jasmine Cui, Charles Ye Categories: cs.CL, cs.LG

Abstract

This paper investigates how language models perform reasoning in hidden continuous space. The authors discover that latent reasoning transformers spontaneously develop structured search behaviors: exploration across candidate answers, tentative selection, and either convergence or reversal (backtracking). Backtracking occurs in approximately one-third of cases and significantly improves accuracy. When distractors are replaced with implausible alternatives, the exploration phase shortens. The key finding is that “latent reasoning models achieve in activation space what chain-of-thought achieves through words: the ability to be wrong, notice, and recover.”

Key Contributions

• Discovery of emergent search: Latent reasoning models spontaneously develop structured search without explicit training for it
• Backtracking mechanism: Models can reverse tentative selections, occurring in ~1/3 of cases with significant accuracy benefits
• Exploration-convergence dynamics: A clear two-phase pattern of exploration followed by convergence or reversal
• Distractor sensitivity: The exploration phase adapts to problem difficulty, shortening when distractors are implausible

Background: Latent Reasoning

Unlike chain-of-thought (CoT) models that reason through explicit token generation, latent reasoning models perform computation in continuous activation space across multiple transformer layers or recurrent steps. The reasoning process is entirely hidden — no intermediate tokens are produced.

This raises a fundamental question: what computational strategies do these models develop internally? Do they simply pattern-match, or do they develop structured problem-solving approaches?

Methodology

Experimental Setup

The authors train latent reasoning transformers on tasks requiring search among candidate answers. They then analyze the internal activations to understand the reasoning process.

Probing Internal States

At each reasoning step $t$, the authors project the model’s hidden state $h_t$ onto the representation space of candidate answers:

$s_i(t) = \text{sim}(h_t, e_i)$

where $e_i$ is the embedding of candidate answer $i$ and $\text{sim}$ is a similarity measure (e.g., cosine similarity). This creates a trajectory showing which candidate the model is “considering” at each step.

Phase Identification

The trajectories reveal distinct phases:

1. Exploration phase: The model’s internal state moves between multiple candidates, with similarity scores fluctuating
2. Selection phase: The model tentatively commits to one candidate, with its similarity score rising
3. Convergence or reversal: Either the selection stabilizes (convergence) or the model abandons it and switches (backtracking)

Key Findings

Emergent Search Structure

Without any explicit training signal for search behavior, the models develop a clear explore-then-exploit pattern:

• Early reasoning steps show high entropy across candidates
• The model systematically evaluates different options
• A tentative selection emerges as one candidate’s similarity score rises above others

Backtracking Behavior

In approximately one-third of cases, the model exhibits backtracking:

1. The model initially commits to candidate $A$ (similarity score rises)
2. At some step $t^*$, the model reverses course
3. The similarity to $A$ drops while similarity to candidate $B$ rises
4. The model ultimately converges on $B$

This backtracking is not random — it occurs when the initial selection was incorrect, and the reversal leads to the correct answer with high probability. The accuracy on backtracking cases is significantly higher than if the model had stuck with its initial selection.

Distractor Effects

When implausible distractors replace plausible ones:

• The exploration phase shortens significantly
• The model spends fewer steps evaluating obviously wrong candidates
• Convergence happens earlier, suggesting the model recognizes answer plausibility

This demonstrates that the exploration is genuinely evaluative, not merely a fixed computational routine.

Comparison with Chain-of-Thought

The paper draws a striking parallel:

Aspect	Chain-of-Thought	Latent Reasoning
Medium	Token space	Activation space
Exploration	Generating candidate reasoning paths	Internal state visiting candidate representations
Backtracking	”Wait, let me reconsider…”	Reversal in similarity trajectories
Error recovery	Explicit self-correction in text	Implicit state reversal

The core insight: latent reasoning models achieve the same functional capabilities as CoT — the ability to explore, make tentative commitments, detect errors, and recover — but entirely within continuous activation space.

Quantitative Analysis

• Backtracking frequency: ~33% of problems exhibit backtracking
• Accuracy with backtracking: Significantly higher than without on the same problem types
• Exploration length: Correlates with problem difficulty
• Convergence stability: Once the model converges after backtracking, it rarely reverses again

Implications

For Interpretability

The discovery of structured search in activation space provides a new lens for understanding latent reasoning:

• Internal computations are not opaque — they follow interpretable patterns
• Probing techniques can reveal the model’s “thought process” even without explicit reasoning tokens
• The exploration-convergence pattern may be a universal feature of learned reasoning

For Model Design

Understanding emergent search suggests design improvements:

• Providing sufficient reasoning depth (layers/steps) is critical for the exploration phase
• Models may benefit from architectures that facilitate backtracking
• The number of reasoning steps needed scales with problem difficulty

For the CoT vs. Latent Reasoning Debate

This work suggests that CoT and latent reasoning are not fundamentally different strategies but rather the same computational pattern expressed in different media. The key capability — being wrong, noticing, and recovering — can emerge in either setting.

Significance

This paper provides compelling evidence that structured reasoning behaviors emerge naturally in latent reasoning models without explicit supervision. The discovery of backtracking in activation space is particularly striking, as it demonstrates that error correction — previously thought to require explicit verbalization — can occur entirely in continuous hidden states. This work bridges the gap between interpretable CoT reasoning and efficient latent computation, suggesting that the benefits of “thinking out loud” can be achieved silently.