From Context Overflow to Learned Compression: How CompactionRL Trains Long-Horizon LLM Agents

Paper: 2607.05378 Authors: Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, Yuxiao Dong Categories: cs.LG

The Gap

Reinforcement Learning (RL) from human feedback or outcomes has become a standard tool for honing LLM capabilities, particularly for agentic tasks like coding. Simultaneously, techniques for “context compaction” – summarizing long histories into shorter ones – exist to manage the finite context window of LLMs. However, the world has treated these as two separate problems: first train the agent, then maybe compact its memory after the fact when it gets stuck. This paper addresses the fundamental disconnect: *how can we teach an agent through compaction, rather than treating compaction as a post-hoc repair? Prior work either ignored compaction in RL (leading to context overflow and training failure) or applied it heuristically outside the learning loop (losing valuable learning signal). The paper’s contribution is closing this loop, making compaction an integrated, learnable part of the RL training process itself.

[Problem]
| Finite context window in LLMs limits
| agentic task completion in long horizons.
v
[Assumption] <--> [Limitation]
| RL optimizes actions,       | Existing compaction is
| not summarization.          | separate & heuristic.
v                             v
[Gap]
| No method to jointly learn
| task execution and context
| compression via RL.
v
[Method: CompactionRL]
| Joint RL objective: optimize
| task reward + summary quality.
| Token-loss normalization &
| cross-trajectory GAE.
v
[Evidence]
| Open models gain 3-7 pp on
| SWE-bench & Terminal-Bench.
| Enables training of much larger
| models (GLM-5.2).
v
[Conclusion]
| Context compaction can and should
| be a learnable skill within the
| RL loop, not an afterthought.

The Increment

One sentence: Before this paper, context compaction was a utility tool applied after RL training; after this paper, it is a core, trainable capability within the RL loop itself, enabling the creation of long-horizon agents that learn to manage their own memory.

Core Mechanism

CompactionRL’s innovation lies in its training objective and data flow. The system treats the LLM as a dual-purpose agent: it must both execute actions (e.g., write code, run terminal commands) and generate summaries of its past context when that context grows too large. These two functions are optimized with a single, unified reinforcement learning signal. The key technical contributions enabling this are:

1. Token-level loss normalization: The reward signal from the task (e.g., “did the code fix the bug?”) is normalized across all tokens, including those in the summary. This prevents the summary generation from dominating the loss just because it uses many tokens.
2. Cross-trajectory Generalized Advantage Estimation (GAE): The RL advantage estimation, which guides the model toward good actions, is computed not just over the current action trajectory but can leverage information from previous, compacted trajectories. This allows the model to learn from the “lessons” summarized from past experiences.

The data flow is cyclical: The agent acts, its context grows, a summary is triggered, the summary is generated by the same LLM being trained, the context is replaced, and the agent continues learning from the new, compacted state. The RL update considers the entire loop.

[LLM Agent]
| (Initial Context)
v
[Execute Action] --> [Observe Result]
^                         |
|                         v
|              [Context Growing?]
|                   |       |
|                  Yes      No --> [Continue Task]
|                   |
|                   v
|         [Generate Summary]
|         (Learned Skill)
|                   |
|                   v
[Update Context] <---| (Replace history with summary)
^                   |
|                   v
+------[RL Loop: Reward + Cross-Trajectory GAE]------+

Structural Metaphor: Imagine training a junior software engineer. Normally, you’d give them a long notebook of past attempts (the context) and hope they finish the task before the notebook fills up. CompactionRL is like training them with two skills simultaneously: how to code (task execution) and how to write concise, useful commit messages and documentation (summary generation). The key is that their “performance review” (the RL reward) evaluates both the final code and the quality of their documentation. Moreover, they are taught to look back at summaries from their old projects (cross-trajectory) to inform new work. They don’t just learn *to code; they learn to code while managing their own knowledge base.

Key Concepts

• Joint Optimization: Traditionally, you train a model for Task A (coding) and separately might use a tool for Task B (summarization). Here, the model’s weights are updated to get better at both A and B *simultaneously, using a single feedback signal from the world. It’s like learning to play basketball by shooting hoops, but your coach only scores you on made baskets—except now, the coach also gives you points for a clean, accurate pass that sets up the shot. The pass (summary) becomes directly optimized for the ultimate goal (basket/task success).
• Token-level Loss Normalization: In a standard RL setup, if your reward is +1 for a correct solution, the model gets that +1 distributed over all the tokens it output. If the summary is very long (say, 1000 tokens) and the final action is short (10 tokens), the summary generation would dominate the learning, as each of its 1000 tokens gets a tiny piece of the reward. This paper scales the reward contribution by token count, ensuring the model pays equal “attention” to learning from the short, crucial final decision and from the long, informational summary.

Framework Shift

Before (mainstream approach):        After (this paper):

[RL Training Loop]                   [RL Training Loop]
     |                                    |
     v                                    v
[Agent Acts]                         [Agent Acts] <------+
[Context Fills]                           |              |
     |                                    v              |
     v                              [Context Fills]?     |
[Fail / Truncate]                         |              |
     |                               Yes--|--No          |
     v                                    v              |
[Manual Heuristic Summary]          [Learned Summary Gen]
(Outside RL)                                |
     |                                    v
     v                              [Compact Context]
[Restart or Give Up]                        |
                                            v
                                   [Continue within same
                                    RL loop, learn from
                                    both execution and
                                    summary via reward]

From treating compaction as an external tool to making it an internal, optimized skill, the core shift is that the agent’s memory management is no longer a liability but a trainable asset.

Expert Assessment

Problem choice: Strong. The context window is a fundamental, practical bottleneck for deploying LLM agents on real, long-horizon tasks. Connecting it to the RL training loop is a logical and timely leap. It sits at the intersection of two hot areas (agentic RL and long-context efficiency) and addresses a real pain point, not a theoretical curiosity.

Method maturity: Clever with some complexity. The joint optimization insight is elegant. The cross-trajectory GAE is a smart way to reuse information, but it adds significant conceptual and implementation complexity. One wonders if a simpler approach—like only summarizing at the very end of a failed trajectory and feeding that summary into the next attempt’s initial prompt—would capture 80% of the gain. The authors don’t deeply explore such simpler baselines, which is a minor weakness.

Experimental integrity: Good, but with caveats. The gains (3-7 points) are solid and consistent across models and two challenging benchmarks. Training on top of open models (GLM-4.5/4.7) is a fair test. However, the absolute numbers (66.8% on SWE-bench Verified) are still far behind state-of-the-art proprietary models, suggesting this technique is more about enabling capability (making long tasks trainable at all) than achieving top-tier performance. The comparison is primarily with the untuned base model, which is the correct comparison for measuring the *increment, but the reader should not misconstrue the results as placing these models at the frontier.

Writing quality: Functional but could be sharper. The motivation and high-level idea are clearly presented. The “Core Mechanism” section, however, dives into technical details (token normalization, GAE) without always first grounding them in the intuitive “why.” The structural metaphor above is absent from the paper; adding a page of such intuition would greatly improve accessibility for the broader ML audience.

Verdict: Weak accept — A clear, valuable contribution that solves a real problem with a non-trivial method. It opens a well-defined research direction (learned context management in RL) even if the specific technique may be iterated upon.

Takeaways

1. Treat Context as a Learnable Resource, Not Just a Fixed Budget: The most transferable idea is to frame information compression itself as a policy to be optimized within any learning system that operates over long sequences. This applies to RL agents, but also to any recurrent model or even dialogue systems.
2. The “Summary-as-Skill” Paradigm: If you’re building a system that must retain information over long interactions, consider making the summarization component a first-class citizen in your training objective, not a post-processing step. Train it with the same end-goal signal as the primary task.
3. Cross-Trajectory Generalization is Powerful: The paper shows that summaries from past episodes can inform and accelerate learning in new ones. Practically, this means persisting and reusing distilled knowledge from past experiences can make your training more sample-efficient, even if the tasks are slightly different.