Language Models Need Sleep: Consolidating Context Through Offline Recurrence

Paper: 2605.26099 Authors: Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti Categories: cs.CL, cs.AI

The Gap

Transformers dominate long-context tasks but hit a wall: attention scales quadratically with sequence length. You can’t just keep extending the key-value cache — memory and compute explode. Prior work tried sparse attention, retrieval augmentation, or hybrid architectures mixing transformers with state-space models (SSMs). But sparse attention loses information, retrieval is brittle, and hybrids still carry the KV cache burden during inference.

The core problem: transformers treat all context equally at inference time, recomputing attention over everything. No mechanism exists to consolidate old context into a compressed form that’s cheap to query but still preserves reasoning capacity.

Problem: Long context → quadratic attention cost
         |
         v
Assumption: Can we compress context offline
            without losing reasoning ability?
         |
         v
Method: Sleep phase = N recurrent passes
        converting context → fast weights (SSM)
         |
         v
Evidence: Synthetic tasks (cellular automata, graph retrieval)
          + math reasoning (GSM8K-style)
         |
         v
Conclusion: Sleep duration N correlates with
            reasoning depth required

The Increment

One sentence: Before this paper, transformers either kept growing their KV cache or lost context; after, they can “sleep” to compress recent context into fast weights, shifting computation offline while preserving wake-time latency.

Core Mechanism

The method introduces a two-phase cycle: wake and sleep. During wake, the model processes new tokens normally using attention, accumulating context in a KV cache. When the cache fills (or at regular intervals), the model enters sleep. During sleep, it performs N recurrent passes over the accumulated context. Each pass updates fast weights in SSM blocks through a learned local rule — think of it as distilling the cache into a compressed state representation. After sleep, the KV cache is cleared, and the model wakes with updated fast weights that encode the consolidated context.

The architecture is a hybrid: attention layers for flexible reasoning during wake, SSM blocks for efficient state compression during sleep. The SSM blocks use a state-space formulation where hidden states evolve recurrently. The learned update rule adjusts these states based on the context being consolidated. Crucially, sleep happens offline — it doesn’t add latency to token generation during wake.

Wake Phase:
  Input tokens → Attention (uses KV cache) → Output
                      |
                      v
                 Cache fills
                      |
                      v
Sleep Phase:
  Cached context → [Recurrent pass 1] → Update SSM fast weights
                → [Recurrent pass 2] → Update SSM fast weights
                → ...
                → [Recurrent pass N] → Update SSM fast weights
                      |
                      v
                 Clear cache
                      |
                      v
  Wake with updated fast weights

Think of it like a student cramming for an exam. During the day (wake), you read new material and take notes (KV cache). At night (sleep), you review your notes multiple times (N recurrent passes), extracting key concepts and connections (fast weights). The next morning, you throw away the notes (clear cache) but retain the distilled understanding (updated SSM state). When a question comes (inference), you answer quickly using the distilled knowledge, not by re-reading all your notes.

The metaphor is load-bearing: the notes are the KV cache (explicit, bulky), the distilled understanding is the fast weights (implicit, compact), and the review process is the recurrent consolidation. Just as reviewing notes multiple times (higher N) helps with harder exams, more sleep passes improve performance on tasks requiring deeper reasoning.

Key Concepts

• Fast Weights: Parameters that change rapidly during inference, unlike the frozen base model weights. In this paper, fast weights are the hidden states in SSM blocks. They’re “fast” because they update during sleep to encode recent context, acting as a compressed memory. Contrast with slow weights (the base transformer parameters, fixed after training). Example: if the model just read “Alice gave Bob a book,” the fast weights might encode the transfer relationship without storing the full sentence.

• State-Space Models (SSMs): A class of sequence models where hidden states evolve recurrently according to linear dynamics. Unlike transformers (which attend to all positions), SSMs maintain a fixed-size state vector that updates at each step. They’re efficient for long sequences because state updates are constant-time. In this paper, SSM blocks provide the substrate for fast weight updates during sleep. Think of SSMs as a conveyor belt: items (tokens) pass through, and the belt’s state (hidden vector) changes incrementally, carrying forward a summary of what’s passed.

• Learned Local Rule: The update mechanism for fast weights during sleep. “Local” means it operates on individual tokens or small windows, not the full context at once. “Learned” means it’s trained end-to-end, not hand-designed. During sleep, the model applies this rule N times over the cached context, refining the fast weights iteratively. Example: the rule might say “if you see a causal relationship, strengthen the connection between entities in the state vector.”

Framework Shift

Before (mainstream approach):        After (this paper):

Inference:                           Inference:
  Token → Attention over             Wake:
          full KV cache →              Token → Attention over
          Output                               short KV cache →
                                               Output
  [Cache grows unbounded]                      |
                                               v
                                          Cache full?
                                               |
                                               v
                                          Sleep (offline):
                                            N recurrent passes →
                                            Update fast weights →
                                            Clear cache
                                               |
                                               v
                                          Resume wake with
                                          compressed state

Cost: O(L^2) per token               Cost: O(L) per token (wake)
      where L = context length             + O(N*L) amortized (sleep)

From unbounded cache growth to periodic consolidation, the core shift is moving from stateless attention (recompute everything) to stateful compression (distill and forget).

Expert Assessment

Problem choice: Real gap. Long-context scaling is a bottleneck for deployment, and existing solutions (sparse attention, retrieval) are band-aids. The biological sleep analogy is cute but not necessary — the core insight is offline consolidation, which is sound. This sits at the intersection of efficiency and capability, a high-value area.

Method maturity: Clever but not fully baked. The learned local rule is underspecified in the abstract — how exactly does it work? Is it gradient-based? Hebbian? The hybrid architecture (attention + SSM) is pragmatic but adds complexity. A simpler baseline would be pure SSMs with periodic resets. The N-pass recurrence is elegant, though it’s unclear if N needs to be fixed or can adapt per task.

Experimental integrity: Synthetic tasks (cellular automata, graph retrieval) are good for controlled analysis but limited in scope. GSM8K-style math reasoning is more realistic, though the abstract doesn’t specify dataset size or baseline details. The claim that “regular transformer as well as SSM-attention hybrid models fail” is strong — need to see if baselines were given equivalent compute budgets. The correlation between sleep duration N and reasoning depth is interesting but could be confounded by total compute.

Writing quality: Abstract is dense and jargon-heavy (“fast weights,” “state-space model blocks,” “learned local rule”) without enough intuition. The sleep metaphor is introduced but not exploited for clarity. The results section is vague (“largest gains on examples that require deeper reasoning”) — quantify “largest” and define “deeper.” The related work comparison (what fails, what doesn’t) needs more precision.

Verdict: weak accept — The core idea (offline consolidation via recurrent passes) is novel and addresses a real problem. The experimental setup is reasonable but not exhaustive. The method’s complexity and underspecified components raise questions about reproducibility and generalization. With clearer exposition and stronger baselines, this could be a strong accept.

Takeaways

Offline consolidation as a design pattern: Don’t process everything online. Identify phases where you can shift computation offline (batch jobs, background threads) to compress state. This applies beyond LLMs — think databases (compaction), caches (eviction policies), or even personal knowledge management (periodic review).

Recurrent refinement: One pass isn’t always enough. If you’re distilling information, multiple passes with a simple rule can outperform a single complex pass. This is the “review your notes multiple times” principle — applicable to training (curriculum learning), data processing (iterative refinement), or even writing (multiple drafts).

Hybrid architectures for efficiency: Combine a flexible but expensive component (attention) with a cheap but limited one (SSM). Use the expensive part sparingly (short contexts) and the cheap part for bulk work (long contexts). This tradeoff pattern shows up in systems design (fast path / slow path), algorithms (quicksort + insertion sort), and more.