Prefix Sliding: Most of a Reasoning Trace Stops Mattering

Paper: 2608.26070 Authors: Niklas Muennighoff, Zhengyang Wang, Zeyi Chen, Weijia Shi, Binyuan Hui, John Yang, Dapeng Jiang, Mika Senghaas, Fares Obeid, Johannes Hagemann, Sami Jaghouar, Ludwig Schmidt, Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, Mike Lewis Categories: cs.AI, cs.CL, cs.LG

The Gap

Test-time scaling works by letting a model think longer on hard problems. The mechanism is that the entire reasoning trace stays in memory under full attention, so every new token attends to every previous one. That is fine for short traces and ruinous for long ones: the memory requirement grows with the length of the trace, and the tasks that most need long thinking are exactly the ones where it becomes prohibitive.

The field has treated this as a cost to be paid. The paper’s opening observation reframes it: most intermediate reasoning tokens lose importance as the model continues reasoning. If that is true, then retaining the whole trace is not paying for capability — it is paying for history that has stopped influencing anything.

That distinction is what makes a cheap intervention possible. The question stops being “how do we afford full attention over longer traces” and becomes “which parts of the trace actually matter”.

   TEST-TIME SCALING AND ITS COST

   hard problem -> let the model reason longer
        |
        v
   full attention: every new token attends to ALL previous
        |
        +-- fine for short traces
        +-- memory grows with trace length
        +-- the tasks that NEED long thinking are exactly
            the ones where this becomes prohibitive
        |
        v
   [OBSERVATION] most intermediate reasoning tokens
                 LOSE IMPORTANCE as reasoning continues
        |
        v
   [GAP] we are paying quadratic cost to retain history
         that has stopped influencing the current step

The Increment

One sentence: Before this paper, long-horizon test-time scaling paid for the whole reasoning trace; after it, keeping only the prefix and a recent window caps memory regardless of trace length — 3x faster with no training and no loss, and enabling scaling past a hundred thousand tokens when trained with RL.

Core Mechanism

The intervention is deliberately simple, and the simplicity is the finding.

Keep the prefix and a recent window; discard the middle. Prefix Sliding drops tokens during reasoning that are neither part of the prefix nor part of a window of the last few thousand tokens. The two retained regions correspond to two different functional roles, and naming them is what justifies the design: the prefix holds the key instructions and available tools — the task specification, which must remain visible; the recent window holds the reasoning the model is currently working on. What gets discarded is the long middle: reasoning that was executed and whose influence on the next step has decayed.

The consequence is the property that matters for deployment: memory is capped regardless of how long the model reasons. This is not a constant-factor improvement, it is a change in the asymptotics. A trace of any length now has a bounded footprint, which is what makes long-horizon scaling affordable rather than merely cheaper.

Three results establish the useful range:

Without any training, Prefix Sliding makes existing models 3x faster while maintaining performance. That it works off the shelf is the strongest practical claim, because it means the observation about decaying importance is not a subtle property that requires the model to be adapted to it. The behaviour was already there.

Training with Prefix Sliding via reinforcement learning achieves better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. This is the second stage: once memory is bounded, the ceiling on trace length moves, and RL can exploit traces that were previously unreachable. Faster inference and a higher capability ceiling are separate benefits, delivered by the same mechanism.

Ablations show it outperforms summarising intermediate tokens or a vanilla sliding window. Both comparisons are the right ones. Summarising is the obvious alternative — compress the middle rather than drop it — and it loses, which suggests the discarded tokens are genuinely not worth even the cost of summarising. A vanilla sliding window discards the prefix too, losing the instructions, and it loses as well. The design occupies the narrow useful point between those two failures.

   PREFIX SLIDING: KEEP TWO REGIONS, DROP THE MIDDLE

   [ PREFIX ]            [ ... DISCARDED ... ]        [ WINDOW ]
   key instructions      executed reasoning            last few
   available tools       whose influence has           thousand
   (must stay visible)   decayed                       tokens
        |                                               |
        +------------------+----------------------------+
                           v
              memory CAPPED regardless of trace length
              (a change in asymptotics, not a constant)
                           |
        +------------------+-------------------+
        v                                      v
   NO TRAINING:                        TRAINING (RL):
   3x faster, performance              scales reasoning
   maintained                          BEYOND 100k tokens
                                       -> better performance
                           |
                           v
   ABLATIONS: beats summarising intermediate tokens,
              beats a vanilla sliding window

Think of it as how you work through a long proof on a whiteboard. You do not keep every intermediate line in front of you — you would run out of board and spend your time re-reading. You keep the problem statement pinned at the top (the prefix, always visible), the lines you are actively working on at the bottom (the window), and you erase the middle. You can erase it precisely because you already used it: if a step mattered to where you are now, its result is already folded into the current lines. Erasing is only dangerous when it removes something you still need, which is why the problem statement stays pinned and the recent working stays up. The paper’s ablation against summarising is the empirical version of a related intuition: if a middle line is not worth keeping, it is also not worth the effort of condensing.

Key Concepts

• Decaying importance of intermediate reasoning: the observation motivating everything. Most tokens stop influencing the current step as reasoning proceeds, so retaining them buys nothing — which turns an apparently unavoidable cost into an avoidable one.
• Capped memory as a change of asymptotics: the footprint becomes bounded regardless of trace length. This is what converts long-horizon test-time scaling from “expensive but possible” into “affordable”, and it is why the same mechanism yields both a speed win and a capability ceiling increase.
• Prefix versus window and why both must stay: the prefix carries the specification, the window carries the active reasoning. Dropping either fails — losing the prefix loses the task, losing the window loses the thought — which is exactly what the vanilla-sliding-window ablation demonstrates.
• Zero-training applicability: working on existing models without adaptation. It means the property was already latent in the models’ behaviour, and the intervention is exploiting it rather than teaching it.

Framework Shift

Before (retain the full trace):
  every token attends to all previous tokens
  memory grows with reasoning length
  -> long thinking is prohibitively expensive
  -> the trace is treated as an asset throughout

After (retain prefix + window):
  prefix (instructions, tools) + last few thousand tokens
  -> memory CAPPED regardless of length
  -> 3x faster with no training, no loss
  -> with RL, scales beyond 100k tokens
  -> beats both summarising and vanilla sliding window

From affording full attention over an ever-growing trace, to keeping only the two parts of the trace that are doing work, the core shift is that reasoning history is not uniformly valuable and the cost of retaining it can be avoided rather than budgeted.

Expert Assessment

Problem choice: Excellent, and it is the right kind of observation — a simple property that was available to anyone and had not been exploited. Test-time scaling’s memory cost has been treated as a structural property of the approach, and the paper shows it was a consequence of an unexamined assumption about the value of old tokens.

Method maturity: The intervention is minimal, which is a virtue here: two retained regions, one discarded middle, no training required for the main claim. The ablations are chosen well and both are informative — losing to a vanilla sliding window confirms the prefix matters, and beating summarisation confirms the discarded tokens are not worth even cheap compression. The obvious open question is how the window size interacts with task type: “the last few thousand tokens” is a fixed budget, and tasks with long-range dependencies within the reasoning would presumably need more.

Experimental integrity: Reporting that the method works with no training is the strongest and most falsifiable claim, and it is a claim about existing models rather than about a trained variant, which means it can be checked quickly. The RL result is a separate claim with a separate benefit, and the paper keeps them distinct rather than averaging them into one headline. The scope is reasoning-trace tasks; whether the same decay holds for agentic traces with interleaved tool calls and external results — where a much older token may be the only record of a critical observation — is a genuinely different setting that the paper does not settle.

Writing quality: The two-region design is stated with the function of each region, which is what makes it memorable: the prefix is the instructions, the window is the current thought. The code release makes it checkable. What would improve it further is a characterisation of the window size’s sensitivity, since that is the one hyperparameter a user must choose.

Verdict: strong accept — a simple, well-ablated intervention that removes a cost the field had accepted as structural, and that improves both throughput and the achievable reasoning ceiling.

Takeaways

• Before optimising the cost of retaining context, check whether all of it is still influencing the output. Here most of a reasoning trace had stopped mattering, and the cost was avoidable rather than inherent.
• Keep the specification and the recent working; drop the executed middle. Discarding is safe when a step’s result is already folded into what you are working on, so protect the parts that are not self-contained.
• Prefer bounded memory over faster growth. Capping the footprint regardless of length is what turns a capability from expensive into available, and it delivered a ceiling increase on top of the speedup.
• Test the cheap configuration before training for it. The 3x speedup required no training, which is worth knowing before investing in an adapted model.