Online Experiential Learning: Teaching LLMs From Their Own Deployment History

Paper: 2603.16856 Authors: Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei Categories: cs.CL

The Gap

LLMs are trained offline — on curated datasets, human annotations, or simulated rollouts — and then frozen at deployment. Once a model is live, every interaction it has, every mistake it makes, every clever shortcut it discovers, vanishes. The model that served user #1,000,000 is identical to the one that served user #1. That’s a massive waste.

Prior work has tried to close this loop in a few ways. RLHF and DPO use human feedback, but require expensive annotation pipelines and don’t scale to continuous deployment. Online RL (like in game-playing agents) can adapt in real time, but it requires the training process to have live access to the environment — which is often impossible when the environment is a user’s private system or a third-party service. Retrieval-augmented approaches can surface past trajectories at inference time, but they don’t actually update the model’s weights, so the “learning” is shallow and context-dependent.

The specific gap: there’s no principled way to take the raw experience a deployed model accumulates and turn it into durable, weight-level improvements — especially when the training process can’t reach back into the deployment environment.

Problem: Deployed LLMs don't learn from experience
    |
    v
Assumption: Raw trajectories contain transferable knowledge,
            but are too noisy to distill directly
    |
    v
Method: Extract -> Accumulate -> Distill (on-policy)
    |
    v
Evidence: Consistent gains over iterations on text games,
          better token efficiency, OOD performance preserved
    |
    v
Conclusion: On-policy context distillation from extracted
            knowledge closes the offline/online gap

The Increment

One sentence: Before this paper, deployment experience was a dead end; after it, there’s a concrete two-stage loop that turns interaction history into weight-level improvements without needing live environment access.

Core Mechanism

OEL has two alternating stages that form a closed loop. In Stage 1 (knowledge extraction), the deployed model collects interaction trajectories from real users. These raw trajectories are messy — they contain failed attempts, redundant steps, and model-specific reasoning that doesn’t generalize. So a knowledge extractor (likely a prompted LLM) processes these trajectories and distills them into compact, transferable “experiential knowledge” — think of it as lessons learned, stripped of the noise.

In Stage 2 (knowledge consolidation), this extracted knowledge is used to update the model’s weights via on-policy context distillation. The key constraint here is that Stage 2 has no access to the original user-side environment. Instead, the extracted knowledge is treated as context, and the model is trained to internalize it — to produce the same outputs it would if it had that knowledge in context, but without needing the context at inference time. This is the “distillation” part: compress the knowledge from context into weights.

The loop then closes: the improved model goes back to deployment, collects better trajectories (because it’s smarter now), which yield richer extracted knowledge, which leads to better distillation in the next round. Each iteration compounds.

  Deployment                    Training
  (user side)                   (model side)
      |                              |
  [Model v_n]                        |
      |                              |
  collects trajectories              |
      |                              |
  [Raw Trajectories]                 |
      |                              |
  [Knowledge Extractor]              |
      |                              |
  [Experiential Knowledge] --------> |
                                     |
                              [On-Policy Context
                               Distillation]
                                     |
                              [Model v_(n+1)] ---> back to deployment

Think of it like a chef who works a busy restaurant service (deployment), then at the end of the night writes up a clean recipe card summarizing what worked (knowledge extraction), and then drills those techniques until they’re muscle memory (distillation). The next service, they’re faster and more precise. The raw chaos of the dinner rush doesn’t go directly into the recipe card — it gets filtered and formalized first. And the recipe card doesn’t stay on the counter during service — it gets internalized so the chef can cook without looking at notes. That’s the whole loop: experience → lesson → muscle memory → better experience.

Key Concepts

• On-policy context distillation: Regular knowledge distillation trains a student model to mimic a teacher’s outputs. Context distillation is a variant where the “teacher” is the same model but with extra context prepended (e.g., “here’s what you learned last week”). The student learns to produce those outputs without the context. “On-policy” means the student being trained is the same model that generated the trajectories — this matters because if you train model A on trajectories from model B, there’s a distribution mismatch that hurts learning. The paper argues this consistency is critical.

• Transferable experiential knowledge: Raw trajectories are like a transcript of someone fumbling through a task. Transferable knowledge is the extracted principle — “when you see X, do Y, not Z.” The extraction step is what separates OEL from naive replay-based methods. The paper shows empirically that extracted knowledge significantly outperforms raw trajectories as a training signal.

• Online learning loop: Not “online” in the web sense, but in the ML sense — the model updates based on data it generates itself during deployment, rather than a fixed offline dataset. The loop structure means improvements compound: better model → better data → better model.

Framework Shift

Before (mainstream approach):        After (this paper):

  [Offline Data]                       [Deployment]
       |                                    |
  [Train Once]                        [Trajectories]
       |                                    |
  [Deploy]                           [Extract Knowledge]
       |                                    |
  [Collect Experience] --> /dev/null   [Distill into Weights]
                                            |
                                       [Better Model]
                                            |
                                       [Back to Deployment]
                                            ^
                                            |_____(loop)

From static offline training to a living feedback loop, the core shift is: deployment experience is no longer discarded, it’s the training signal.

Expert Assessment

Problem choice: This is a real gap, not a manufactured one. The “deployed models don’t learn” problem is obvious to anyone who’s run a production LLM system. The specific contribution — handling the case where training can’t access the deployment environment — is a practically important constraint that prior work glosses over. The field is clearly moving toward continual learning for LLMs, and this paper is a reasonable early stake in the ground.

Method maturity: The two-stage design is clean and the on-policy consistency argument is well-motivated. That said, the knowledge extraction step is doing a lot of heavy lifting and relies on a prompted LLM — which means the quality of extracted knowledge depends on the extractor’s capability, and the paper doesn’t deeply analyze failure modes here. It’s a clever insight, but the extraction step feels underspecified. A skeptic could ask: isn’t this just “use a strong LLM to summarize your logs and fine-tune on the summaries”? The answer is yes, roughly, but the on-policy distillation framing and the iterative loop are the real contributions.

Experimental integrity: Text-based games are a reasonable testbed — they have clear reward signals and controllable complexity. Testing across multiple model scales and thinking/non-thinking variants is good practice. The OOD evaluation is a nice touch; many continual learning papers ignore catastrophic forgetting entirely. That said, text games are a narrow domain. The paper would be significantly stronger with one real-world deployment scenario (even a synthetic one that’s more realistic than a game). The baselines seem fair but the paper would benefit from a stronger ablation on the extractor quality.

Writing quality: The paper is technically clear but the knowledge extraction step — arguably the most novel and fragile part — gets the least scrutiny. The section describing how extraction actually works (what prompts, what format, what failure modes) reads like it was written last and under time pressure. Rewriting that section with concrete examples of raw trajectory → extracted knowledge would make the whole paper more convincing.

Verdict: weak accept — solid problem, clean framing, but the extraction mechanism needs more rigorous treatment and the evaluation domain is too narrow to fully trust the generalization claims.

Takeaways

A few concrete things worth stealing:

1. The “extract then distill” pattern is broadly applicable. If you have noisy demonstration data (from users, from simulations, from web scraping), don’t fine-tune on it raw — run it through an extraction step first to get clean, generalizable principles. This is underused in practice.

2. On-policy consistency as a design constraint. When building any self-improvement loop, make sure the model generating the training signal is the same model (or a close ancestor) being trained. Distribution mismatch between data source and policy is a silent killer in these setups.

3. The iterative loop structure — where each round’s improved model generates better training data for the next round — is a general recipe for compounding gains in any domain where you can collect deployment feedback. The paper gives you a concrete instantiation to adapt.