KV-Fold: Treating KV Cache as a Fold Accumulator for Long-Context Inference

Paper: 2605.12471 Authors: Alireza Nadali, Patrick Cooper, Ashutosh Trivedi, Alvaro Velasquez Categories: cs.LG, cs.AI, cs.CL

The Gap

Pretrained transformers have fixed context windows (typically 4K-8K tokens). When you need to process longer sequences, you hit a wall: either you fine-tune the model on longer contexts (expensive, requires retraining), or you use streaming methods that discard old information to stay within memory limits (loses long-range dependencies). The field has been stuck between two bad options: retrain everything or accept information loss.

Prior work like StreamingLLM keeps only recent tokens plus a few “attention sinks,” sacrificing exact retrieval for bounded memory. Other approaches require architectural modifications or additional training. The gap: can a frozen pretrained transformer handle arbitrarily long contexts without losing information and without retraining?

Problem: Fixed context window (4K-8K tokens)
   |
   v
Existing solutions:
   |
   +---> Fine-tune on long contexts ---> Expensive, requires retraining
   |
   +---> Streaming (discard old tokens) ---> Loses long-range info
   |
   v
Gap: Can frozen models handle long contexts without loss?
   |
   v
Assumption: KV cache can act as recurrent state
   |
   v
Method: KV-Fold (chunk + accumulate + repeat)
   |
   v
Evidence: 100% retrieval @ 128K tokens, stable across 511 steps
   |
   v
Conclusion: Frozen transformers already support stable KV recurrence

The Increment

One sentence: Before this paper, long-context inference required either retraining models or accepting information loss; after, you can process arbitrarily long sequences with frozen models by treating the KV cache as a fold accumulator.

Core Mechanism

KV-Fold splits the input sequence into chunks and processes them sequentially. At each step, the model receives two inputs: the current chunk and the accumulated KV cache from all previous chunks. The model attends to both, generates new keys and values for the current chunk, concatenates them to the accumulated cache, and passes the enlarged cache to the next step. This is repeated for all chunks.

The key insight is repurposing KV cache concatenation—originally designed for multi-agent communication—as a recurrence mechanism. When processing chunk t, the model sees the KV cache as a prefix context. The cache acts as compressed memory of everything seen so far. No weights change, no special tokens are added, no architectural modifications are made.

Input sequence: [chunk_1][chunk_2][chunk_3]...[chunk_n]

Step 1:
  Input: chunk_1
  KV_cache: empty
  Output: KV_1
  
Step 2:
  Input: chunk_2 + KV_1 (as prefix)
  Model attends to: [KV_1 | chunk_2]
  Output: KV_2 = concat(KV_1, new_keys_values_2)
  
Step 3:
  Input: chunk_3 + KV_2 (as prefix)
  Model attends to: [KV_2 | chunk_3]
  Output: KV_3 = concat(KV_2, new_keys_values_3)
  
...repeat until all chunks processed

Think of it like a snowball rolling down a hill. The snowball is the KV cache. As it rolls through each section of the hill (a chunk), it picks up new snow (new keys and values) and grows larger. The snowball carries everything it has collected so far. When it reaches the next section, it’s already carrying the compressed memory of all previous sections. The hill doesn’t change, the snowball doesn’t change its fundamental structure—it just accumulates. At any point, you can examine the snowball and see the accumulated effect of every section it has rolled through. The model is the hill’s physics: it determines how new snow sticks to the snowball, but the physics themselves don’t change from section to section.

Key Concepts

• KV Cache as Recurrent State: In standard transformer inference, the KV cache is just an optimization to avoid recomputing attention for past tokens. KV-Fold elevates it to a first-class recurrent state. Instead of discarding the cache after processing a sequence, you carry it forward as compressed memory. When the model processes the next chunk, it attends to this cache as if those past tokens were still present. The cache becomes the hidden state in a recurrent architecture, but without modifying the transformer itself. Concrete example: processing a 64K token document in 4K chunks means the model at chunk 16 is attending to a KV cache representing the previous 60K tokens, even though it only sees 4K tokens directly.

• Left Fold Over Chunks: In functional programming, foldl takes a list, an accumulator, and a function, then applies the function to each element and the accumulator sequentially. KV-Fold does exactly this: the list is the sequence of chunks, the accumulator is the KV cache, and the function is the transformer’s forward pass. At each step, you take the current cache (accumulator), process the next chunk (list element), and produce an updated cache (new accumulator). The same operation repeats for every chunk. This is why it’s called KV-Fold—it’s literally a fold operation where the transformer is the combining function.

• Numerical Stability Plateau: The authors found that per-step drift (how much the cache changes at each step) rises briefly at the start, then flattens into a stable plateau that persists across hundreds of steps. This plateau is insensitive to precision changes (float32 vs float16 vs bfloat16), robust across chunk sizes, and consistent across model families. This is surprising because recurrent systems often suffer from vanishing or exploding gradients. Here, the drift saturates naturally, suggesting the pretrained transformer’s attention mechanism already has built-in stabilization. It’s like a self-regulating system that finds an equilibrium without external intervention.

Framework Shift

Before (streaming methods):          After (KV-Fold):

Input: [====long sequence====]       Input: [====long sequence====]
         |                                    |
         v                                    v
    Sliding window                       Split into chunks
         |                                    |
    [--window--]                        [c1][c2][c3]...[cn]
         |                                    |
    Discard old tokens                       v
         |                              Accumulate KV cache
         v                                    |
    Bounded memory,                     KV_0 -> KV_1 -> KV_2 -> ...
    lost information                         |
                                             v
                                        Full memory,
                                        no information loss

From sliding windows that discard history to accumulating caches that preserve it, the core shift is treating memory as something to grow rather than bound.

Expert Assessment

Problem choice: This is a real gap. Long-context inference is a bottleneck for deploying LLMs in production, and existing solutions (fine-tuning, streaming) have clear downsides. The problem sits at the intersection of efficiency and capability—a sweet spot for practical impact. The framing as a functional programming primitive (fold) is clever and positions the work as discovering latent structure rather than inventing new mechanisms.

Method maturity: This is more insight than invention. The authors didn’t design a new architecture—they discovered that concatenating KV caches, a primitive already used for multi-agent communication, can be repurposed for long-context recurrence. The simplicity is a strength: no training, no architectural changes, just a different way of using existing machinery. However, the method is essentially sequential processing with growing memory, which limits parallelism. The stability findings are the real contribution—showing that this naive approach doesn’t collapse is non-obvious and valuable.

Experimental integrity: The needle-in-a-haystack benchmark is clean and the 100% retrieval result is strong. However, this is a synthetic task designed to test exact information retention. Real-world long-context tasks (summarization, reasoning over documents) are not evaluated. The stability analysis is thorough (10,000x precision change, multiple models, varying chunk sizes), which builds confidence. The comparison to streaming methods is fair but limited—no comparison to other long-context methods like RoPE scaling or attention sinks with larger windows. The memory analysis is honest: they acknowledge the cache grows linearly with sequence length, which is a fundamental limitation.

Writing quality: The paper is well-structured and the functional programming framing is effective. The stability analysis is the strongest section—clear metrics, thorough ablations, and honest discussion of limitations. The related work section could be deeper: the connection to state-space models and other recurrent architectures is mentioned but not explored. The results section focuses heavily on needle-in-a-haystack, which is a narrow evaluation. A section on failure modes or tasks where KV-Fold struggles would strengthen the paper.

Verdict: weak accept — The method is simple and practical, the stability findings are surprising and well-documented, but the evaluation is narrow and the fundamental memory scaling limitation is not addressed.

Takeaways

The functional programming lens (fold as a design pattern) is transferable. When you have a sequential process with accumulating state, framing it as a fold makes the structure explicit and suggests optimizations (parallelizing independent folds, checkpointing accumulators). The stability plateau finding suggests that pretrained transformers have more robust recurrent properties than expected—this could inform future architecture design. For practitioners: if you need exact long-context retrieval and can tolerate growing memory, KV-Fold is a zero-cost baseline that requires no retraining. The chunk size vs memory tradeoff is explicit and tunable. The method won’t replace fine-tuned long-context models for complex reasoning tasks, but it’s a strong fallback when retraining isn’t an option.