Fast-Slow Learning: Teaching LLMs to Adapt Without Forgetting

Paper: 2605.12484 Authors: Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri Categories: cs.LG, cs.AI

The Gap

Current LLM adaptation sits at an uncomfortable fork. Parameter updates (like RL fine-tuning) achieve strong task performance but cause catastrophic forgetting—the model overwrites general capabilities with task-specific patterns. In-context learning preserves the base model but hits a performance ceiling because it can’t absorb complex task structure into the model itself. Researchers have treated this as a binary choice: either update weights and risk forgetting, or freeze weights and accept limited adaptation.

The gap is architectural: we lack a mechanism that lets LLMs learn at multiple timescales simultaneously—fast adaptation to task specifics without slow drift from general capabilities.

Problem: Binary choice between adaptation methods
   |
   v
Observation: Humans learn at multiple timescales (System 1/2)
   |
   v
Hypothesis: LLMs need dual-speed learning channels
   |
   v
Method: Fast weights (optimized context) + Slow weights (parameters)
   |
   v
Evidence: 3x sample efficiency, 70% less KL drift, preserved plasticity
   |
   v
Conclusion: Decoupling learning speeds prevents forgetting while enabling adaptation

The Increment

One sentence: Before this paper, LLMs chose between fast forgetting (parameter updates) or slow learning (in-context only); after, they can learn task-specific patterns rapidly in context while keeping parameters stable for general reasoning.

Core Mechanism

Fast-Slow Training (FST) splits learning into two channels operating at different speeds. The “slow” channel updates model parameters via standard RL, but with reduced learning rate and regularization to keep them anchored near the base model. The “fast” channel optimizes a text prompt (the context) that gets prepended to every input, treating this prompt as a differentiable parameter that can absorb task-specific patterns.

During training, both channels receive the same reward signal from the task. The prompt optimizer uses textual feedback—converting reward signals into natural language critiques—to update the context through gradient-based prompt tuning. The parameter optimizer uses standard policy gradient methods but with KL penalties that resist drift from the base model. The key is the division of labor: fast weights (context) capture task-specific heuristics and patterns, while slow weights (parameters) maintain general reasoning capabilities.

At inference, the optimized prompt is simply prepended to new inputs. The model sees task-specific guidance in its context window while its parameters remain close to their pretrained state. When switching tasks, you can swap out the fast weights (the prompt) while keeping the slow weights (parameters) intact, enabling rapid adaptation without retraining.

Training Loop:
                    Reward Signal
                         |
                         v
        +----------------+----------------+
        |                                 |
        v                                 v
   Fast Channel                      Slow Channel
   (Prompt Opt)                      (Param RL)
        |                                 |
        v                                 v
   Text Feedback                    KL Penalty
   Gradient Update                  + Policy Grad
        |                                 |
        v                                 v
   Optimized Context  <----------->  Updated Params
   (Task-specific)                   (General reasoning)
        |                                 |
        +----------------+----------------+
                         |
                         v
                   Combined Model

Think of it like a chef’s kitchen. The slow weights are your fundamental knife skills and understanding of heat, salt, acid—techniques that took years to develop and apply across all cuisines. The fast weights are the recipe card for tonight’s dish: specific ingredient ratios, timing notes, plating instructions. When a new order comes in, you don’t retrain your knife skills; you swap the recipe card. The recipe absorbs dish-specific details (sear the scallops 90 seconds per side, deglaze with white wine) while your foundational skills remain stable. If you tried to encode every recipe into muscle memory (parameter-only learning), you’d start confusing techniques across dishes. If you only read recipes without practice (context-only learning), you’d never develop the judgment to adjust on the fly. FST gives you both: stable skills plus swappable recipes.

Key Concepts

• Catastrophic Forgetting: When you fine-tune a neural network on task B after training on task A, it often “forgets” how to do task A—performance collapses. This happens because the network has limited capacity, and new patterns overwrite old ones. Imagine learning French after Spanish: if every French word replaced a Spanish word in your memory rather than coexisting, you’d lose Spanish as you gained French. In LLMs, fine-tuning on code generation can degrade the model’s ability to write essays, even though these skills should be independent. The problem is that parameters are shared across all tasks, so updating them for one task corrupts representations needed for others.

• Plasticity Loss: A network’s ability to learn new patterns degrades after extensive training—it becomes “rigid.” Early in training, the network easily forms new representations; later, it gets stuck in local optima and resists change. Think of a hiking trail: the first few people create a faint path, but after thousands of hikers, the trail is so worn that it’s hard to deviate even slightly. In continual learning, a model trained on task A may struggle to adapt to task B not because it forgot A, but because its parameters have lost the flexibility to encode new patterns. FST preserves plasticity by keeping parameters close to their pretrained state, where they retain the capacity to form new representations.

• Fast vs Slow Weights: This borrows from neuroscience’s distinction between working memory (fast, temporary) and long-term memory (slow, persistent). Fast weights change rapidly to capture immediate context; slow weights change gradually to encode stable patterns. In FST, the prompt is a fast weight—it updates quickly to absorb task-specific heuristics. Parameters are slow weights—they update conservatively to maintain general capabilities. The asymmetry is deliberate: you want task-specific information in a form that’s easy to swap out (the prompt), and general reasoning in a form that’s stable across tasks (the parameters).

Framework Shift

Before (Parameter-only RL):          After (Fast-Slow Training):

Task A Training:                     Task A Training:
  Params ----[RL]----> Params'         Params ----[Slow RL]----> Params'
    |                    |               |                          |
    |                    |               +--[Fast Opt]---> Prompt_A
    v                    v               |                          |
  General            Task A              v                          v
  Reasoning          Specific          General                Task A
                     (Forgot            Reasoning              Specific
                      General)          (Preserved)            (In Context)

Task B Training:                     Task B Training:
  Params' ---[RL]----> Params''        Params' ---[Slow RL]----> Params''
    |                    |               |                          |
    |                    |               +--[Fast Opt]---> Prompt_B
    v                    v               |                          |
  Task A             Task B              v                          v
  Specific           Specific          General                Task B
  (Forgot A)         (Forgot            Reasoning              Specific
                      General)          (Preserved)            (In Context)

One sentence: From monolithic parameter updates that conflate task-specific and general knowledge, to decoupled learning channels where context absorbs task patterns while parameters maintain reasoning stability.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Continual learning and catastrophic forgetting have been open problems for decades, and the LLM era has made them more acute—we’re now fine-tuning billion-parameter models on narrow tasks and watching general capabilities degrade. The framing through dual-process theory (System 1/2) is a bit hand-wavy, but the core observation is sound: current methods force a false dichotomy between adaptation speed and stability.

Method maturity: The core idea—optimizing prompts alongside parameters—is elegant but not entirely novel. Prompt tuning and soft prompts have been explored extensively. What’s new here is the explicit framing as a dual-speed system with different update rates and the empirical demonstration that this prevents forgetting in continual learning. The method is more “clever combination” than “fundamental insight.” That said, the execution is solid: they use textual feedback for prompt optimization (not just gradient descent on embeddings), which makes the fast weights more interpretable.

Experimental integrity: The baselines are fair—they compare against parameter-only RL, context-only learning, and ablations. The tasks (reasoning benchmarks like GSM8K, ARC) are standard but somewhat narrow; it would be stronger to see results on more diverse task distributions. The KL divergence measurements are a good proxy for drift, but they don’t directly measure retention of specific capabilities (e.g., does the model still write good poetry after training on math?). The continual learning experiments are the most compelling: FST continues to learn new tasks while parameter-only RL stalls. One red flag: the paper doesn’t deeply explore failure modes—when does the fast channel hit capacity? How does performance degrade as you add more tasks?

Writing quality: The paper is well-structured and the motivation is clear. The weakness is in the mechanistic details: the prompt optimization procedure is under-explained (what exactly is “textual feedback”? how is it converted to gradients?). Section 3.2 would benefit from a concrete example showing the prompt before and after optimization. The related work section is thorough but could be tighter—some citations feel like box-checking rather than building a narrative.

Verdict: weak accept — Solid empirical work on an important problem, but the method is more incremental than transformative. The continual learning results are strong enough to warrant publication, but the lack of mechanistic depth and limited task diversity keep it from being a strong accept.

Takeaways

Decouple learning rates by information type: If you’re fine-tuning a model, separate task-specific patterns (which should update quickly) from general capabilities (which should update slowly). This doesn’t require FST’s exact architecture—you could use layer-wise learning rates, adapter modules, or even manual prompt engineering. The key is recognizing that not all knowledge should be learned at the same speed.

Use context as a differentiable hyperparameter: Instead of treating prompts as fixed strings, optimize them as part of the training loop. This is especially useful when you have a small amount of task-specific data but want to preserve a pretrained model’s general capabilities. The prompt becomes a lightweight task adapter.

Measure drift, not just performance: When evaluating fine-tuning, track KL divergence from the base model or performance on held-out general tasks. A model that gets 95% on your target task but forgets how to do basic reasoning is less useful than one that gets 90% while retaining general capabilities. FST’s emphasis on drift metrics is a good reminder that task performance alone is an incomplete picture.

Continual learning as a stress test: If your adaptation method works for one task but fails when you add a second, it’s not truly solving the adaptation problem—it’s just overfitting. Use multi-task or continual learning scenarios as a litmus test for whether your method is learning reusable patterns or just memorizing task-specific hacks.