Reasoning in Memory: Teaching LLMs to Think Without Speaking

Paper: 2605.30343 Authors: Lukas Aichberger, Sepp Hochreiter Categories: cs.CL, cs.AI

The Gap

Current methods scale test-time reasoning by generating intermediate tokens—chain-of-thought prompting, self-consistency, tree search over reasoning paths. This works, but it conflates two distinct activities: internal computation (figuring out the answer) and external communication (showing your work). Every reasoning step burns autoregressive generation cost. The model must “think out loud” even when the intermediate steps add no value to the final answer.

Prior latent reasoning attempts (hidden states, learned embeddings) struggled because they lacked a training curriculum to ground the latent representations. The gap: no method successfully decouples reasoning from token generation while maintaining competitive performance.

Problem: Reasoning = Autoregressive Generation
   |
   v
Assumption: LLMs have latent working memory capacity
   |
   v
Method: Fixed memory blocks + two-stage curriculum
   |
   v
Evidence: Matches CoT performance, 1 forward pass vs N
   |
   v
Conclusion: Reasoning can be internalized without generation

The Increment

One sentence: Before—reasoning required generating O(N) intermediate tokens autoregressively; after—reasoning happens in fixed memory blocks processed in O(1) forward pass.

Core Mechanism

RiM introduces memory blocks: fixed sequences of special tokens (e.g., <mem1>, <mem2>, …, <mem32>) inserted between the question and answer. These aren’t generated—they’re prepended to the input. The model processes them in a single forward pass, using them as scratch space for internal computation.

Training happens in two stages. Stage 1 (grounding): After each memory block, the model predicts an explicit reasoning step (like chain-of-thought). This teaches the model to use memory blocks for structured thinking. Stage 2 (refinement): Drop the intermediate supervision. After each memory block, the model predicts the final answer. Early blocks produce rough answers; later blocks refine them. The model learns to iteratively improve its answer using memory as working space.

Input:  [Question] <mem1> <mem2> ... <mem32> [Answer]
         |          |      |           |        |
         v          v      v           v        v
Stage 1: Q ------> Step1  Step2  ...  StepN -> A
         (ground memory blocks to explicit reasoning)

Stage 2: Q ------> Ans1   Ans2   ...  AnsN  -> A
         (refine answer iteratively, no intermediate steps)

Think of it like a student solving a math problem on scratch paper. Traditional chain-of-thought is like narrating every calculation out loud—“first I multiply 7 by 3, which gives 21, then I add 5…”—expensive and slow. RiM is like working silently on scratch paper, showing only the final answer. The memory blocks are the scratch paper: fixed space where the model can scribble intermediate thoughts without the cost of generating them token-by-token. Stage 1 teaches the student how to use scratch paper (by checking their work at each step). Stage 2 lets them work freely, only checking the final answer.

Key Concepts

• Working Memory vs Communication: Human cognition separates internal manipulation (working memory) from external expression (speech). You can solve “17 × 23” in your head without verbalizing every step. LLMs traditionally lack this separation—every computation must be externalized as tokens. RiM introduces the separation by giving models a fixed internal workspace (memory blocks) distinct from the output stream. The model learns to “think” in memory blocks and “speak” only the final answer.

• Curriculum Grounding: You can’t just tell a model “use these special tokens for reasoning”—it has no prior about what they mean. Stage 1 grounds them by forcing the model to predict explicit reasoning steps after each memory block. This creates a bridge: memory blocks → reasoning steps → answer. Once grounded, Stage 2 removes the bridge (no more intermediate steps) but the model retains the learned association between memory blocks and structured reasoning. It’s like training wheels: necessary at first, removed once the skill is internalized.

• Iterative Refinement: Instead of generating a single reasoning chain, RiM predicts the final answer multiple times—once after each memory block. Early predictions are rough; later ones incorporate more computation. This mirrors how humans refine answers: first approximation, then correction, then polish. The loss function penalizes all intermediate answers, so the model learns to use each memory block to improve on the previous attempt.

Framework Shift

Before (Chain-of-Thought):          After (RiM):

Q --> [Generate] --> Step1          Q + <mem1><mem2>...<memN>
      [Generate] --> Step2            |
      [Generate] --> Step3            v
      [Generate] --> StepN          [Single Forward Pass]
      [Generate] --> Answer           |
                                      v
(N autoregressive steps)            Answer

Cost: O(N) generation               Cost: O(1) forward pass
Reasoning: externalized             Reasoning: internalized

From sequential generation to parallel processing, the core shift is decoupling computation from communication.

Expert Assessment

Problem choice: Real gap. Test-time compute scaling is a hot topic (o1, self-consistency, tree search), but all current methods pay the autoregressive tax. Decoupling reasoning from generation is a natural next step, though prior attempts (learned embeddings, hidden state manipulation) failed to gain traction. This paper’s timing is good—it addresses a bottleneck in a high-value area.

Method maturity: The two-stage curriculum is clever and well-motivated. Stage 1 solves the grounding problem that plagued earlier latent reasoning work. Stage 2’s iterative refinement is elegant—it reuses the same mechanism (predict answer after each block) rather than introducing new components. However, the method is somewhat brute-force: fixed memory blocks of arbitrary length (32 tokens) with no learned allocation. A more sophisticated approach might dynamically allocate memory or learn when to stop refining. The paper doesn’t explore these directions.

Experimental integrity: Baselines are fair—CoT, self-consistency, and prior latent methods. Results show RiM matches or slightly exceeds CoT on GSM8K, MATH, and ARC while using one forward pass instead of N. The ablations are thorough (varying memory block count, comparing stage 1 vs stage 2). One concern: the experiments use relatively small models (up to 7B parameters). It’s unclear whether the method scales to frontier models (70B+) where autoregressive cost matters most. The paper also doesn’t report wall-clock time comparisons—“one forward pass” sounds fast, but 32 extra tokens still add compute.

Writing quality: The paper is clear and well-structured. The motivation (working memory analogy) is compelling. However, Section 3.2 (training procedure) is dense—it conflates curriculum design, loss functions, and implementation details. Splitting this into “Curriculum Design” and “Training Mechanics” would improve readability. The related work section undersells the novelty—it lists prior latent reasoning methods but doesn’t clearly explain why they failed and how RiM differs.

Verdict: weak accept — Solid contribution with a clever training curriculum, but limited exploration of design space and unclear scaling to large models.

Takeaways

Curriculum grounding for latent representations: If you want a model to use special tokens or embeddings for internal computation, don’t just add them and hope. Ground them first with explicit supervision (Stage 1), then remove the scaffolding (Stage 2). This two-stage pattern transfers to other domains—teaching models to use tool APIs, intermediate data structures, or planning tokens.

Iterative refinement as a training objective: Instead of predicting the answer once, predict it multiple times at different “depths” of computation. Penalize all predictions, not just the final one. This encourages the model to progressively improve its answer, which is useful for any task where reasoning depth varies (math, coding, planning).

Fixed vs generated tokens for compute scaling: When scaling test-time compute, consider whether intermediate tokens need to be generated or can be fixed. If the structure is predictable (e.g., “think for N steps”), fixed tokens avoid autoregressive cost. This applies beyond reasoning—fixed tokens could represent planning steps, retrieval queries, or tool calls.