Attractor Models: Making Iterative Refinement Scalable Through Fixed-Point Learning

Paper: 2605.12466 Authors: Jacob Fein-Ashley, Paria Rashidinejad Categories: cs.LG, cs.AI, cs.CL, cs.NE

The Gap

Looped Transformers promise iterative refinement for better reasoning, but they hit three walls: training instability from backpropagating through many recurrent steps, memory cost that scales linearly with loop depth, and the need to pick a fixed recurrence depth upfront. Prior work either uses shallow loops (limiting expressiveness) or deep loops (exploding memory and gradients). The field has been stuck choosing between stability and depth.

Problem: Recurrent models need depth but can't afford it
   |
   v
Assumption: What if we never unroll the loop explicitly?
   |
   v
Method: Solve for fixed point implicitly, differentiate through equilibrium
   |
   v
Evidence: 46.6% perplexity gain, 91.4% on Sudoku-Extreme with 27M params
   |
   v
Conclusion: Recurrence becomes learnable computation, not architectural burden

The Increment

One sentence: Before this paper, iterative refinement meant choosing between shallow loops (weak) or deep loops (expensive); after, models learn to internalize iteration into a single equilibrium-seeking step with adaptive depth and constant memory.

Core Mechanism

Attractor Models split computation into two phases. First, a backbone module (standard Transformer) produces an initial output embedding. Second, an attractor module treats this embedding as the starting point of a dynamical system and solves for its fixed point—the state where further iteration produces no change. Instead of unrolling T explicit recurrent steps and backpropagating through all of them, the model uses implicit differentiation: compute gradients at the converged equilibrium without storing intermediate states.

The attractor module is a small network (often just 2-4 layers) that takes the current embedding and outputs a refined version. At training time, a root-finding solver (like Anderson acceleration or Broyden’s method) iterates this module until convergence, typically 5-20 steps. Crucially, gradients flow backward through the equilibrium condition itself, not through the solver’s trajectory. This keeps memory constant regardless of how many iterations the solver takes.

Input tokens
     |
     v
[Backbone Transformer] -----> initial embedding z_0
     |                              |
     |                              v
     |                    [Attractor Module]
     |                         |        ^
     |                         |        |
     |                         v        |
     |                    z_t+1 = f(z_t) (iterate until z* = f(z*))
     |                              |
     |                              v
     +------------------------> fixed point z*
                                    |
                                    v
                              Output logits

Think of it like a sculptor and a polisher. The backbone is the sculptor—it quickly shapes the rough form from raw marble (input tokens). The attractor is the polisher—it takes that rough form and iteratively smooths it until no more refinement is needed, stopping automatically when the surface stops changing. Traditional looped models are like telling the polisher “make exactly 10 passes,” wasting effort if it converges in 5 or failing if it needs 15. Attractor Models instead say “polish until smooth,” and the sculptor learns to carve forms that need less polishing over time. Eventually, the sculptor gets so good that the polisher barely needs to run—this is equilibrium internalization.

Key Concepts

• Fixed Point: Imagine a function that takes a number and outputs a new number. A fixed point is an input where the output equals the input—applying the function again changes nothing. For example, if f(x) = x/2 + 1, then x=2 is a fixed point because f(2) = 2/2 + 1 = 2. In Attractor Models, the attractor module is this function, and the fixed point is the “final” embedding where further refinement is unnecessary. The model learns to make its attractor module have a useful fixed point for each input.

• Implicit Differentiation: Normally, to compute gradients through a loop, you store every intermediate state (memory scales with loop depth). Implicit differentiation is a mathematical trick: if you know the final equilibrium satisfies z** = f(z*), you can compute gradients with respect to z* by differentiating this equilibrium equation directly, without storing the path taken to reach z*. It’s like knowing the summit of a mountain and computing the slope there, without recording every step of the climb.

• Equilibrium Internalization: During training, the model sees many examples where the backbone’s initial output gets refined by the attractor. Over time, the backbone learns to produce outputs already near equilibrium—the attractor has less work to do. Eventually, the backbone’s output is so close to the fixed point that you can skip the attractor entirely at inference with minimal accuracy loss. The model has “internalized” the iterative refinement into its feedforward pass.

Framework Shift

Before (Looped Transformers):        After (Attractor Models):

Input                                Input
  |                                    |
  v                                    v
[Layer 1] --+                        [Backbone] --> z_0
  |         |                          |
  v         |                          v
[Layer 2]   | (unroll T times)       [Attractor: solve z*=f(z*)]
  |         |                          |
  v         |                          v
[Layer 3] <-+                        Output
  |
  v
Output

Memory: O(T * model_size)            Memory: O(model_size)
Depth: Fixed T                       Depth: Adaptive (converges when ready)
Gradients: Through T steps           Gradients: Through equilibrium

One sentence: From explicit recurrence with fixed depth and linear memory cost, to implicit fixed-point solving with adaptive depth and constant memory.

Expert Assessment

Problem choice: Real gap. Looped models have been a known pain point—everyone wants iterative refinement for reasoning, but training instability and memory costs have kept them niche. This paper attacks the right bottleneck at the right time, as reasoning tasks gain prominence.

Method maturity: Clever insight, not brute force. Implicit differentiation for fixed points is borrowed from optimization theory (Deep Equilibrium Models, 2019), but applying it to language modeling with the backbone/attractor split is novel. The equilibrium internalization phenomenon is genuinely surprising and suggests the method taps into something fundamental about how models learn to reason. However, the paper could explore failure modes more—what happens when the fixed point doesn’t exist or the solver doesn’t converge?

Experimental integrity: Strong. Baselines are fair (standard Transformers, Universal Transformers, looped variants). The Sudoku and maze results are striking—91.4% on Sudoku-Extreme with 27M params where GPT-4 and Claude fail is a clean win. The language modeling results (46.6% perplexity improvement, 770M model beating 1.3B Transformer) are compelling. One concern: the paper doesn’t report wall-clock training time. Implicit differentiation adds solver overhead—how much slower is training in practice?

Writing quality: Mostly clear, but the implicit differentiation section (likely in the appendix) needs to be in the main text with a worked example. Practitioners will struggle to implement this without understanding the gradient computation. The equilibrium internalization section is well-written and carries the paper’s “wow” factor.

Verdict: strong accept — Solves a real problem with a principled method, delivers strong empirical results across two distinct regimes (language modeling and reasoning), and reveals a novel training phenomenon that opens new research directions.

Takeaways

For practitioners: The backbone/attractor split is the transferable idea. You don’t need implicit differentiation to benefit—just train a small refinement module on top of your main model and see if it learns useful iteration. If it does, then invest in the fixed-point machinery.

For researchers: Equilibrium internalization suggests that models can “compile” iterative algorithms into feedforward computation during training. This might apply beyond language—could vision models internalize iterative segmentation? Could RL policies internalize planning loops?

For infrastructure: Implicit differentiation libraries (e.g., DEQ implementations) are mature enough to use. The memory savings are real—if you’re training large models with recurrence, this is worth prototyping.

Concrete technique: When debugging convergence, the paper uses Anderson acceleration with a small history window (m=5). This is more stable than naive fixed-point iteration and faster than full Newton methods. Steal this for any iterative solver in your pipeline.