CoinRAG: Slicing the KV Cache Down to Nuggets

Paper: 2608.07458 Authors: Gyuwan Kim, Cheoneum Park, Tao Yang Categories: cs.CL, cs.AI, cs.IR, cs.LG

The Gap

RAG serving has a well-known bottleneck: the retrieved context is long, and prefill cost grows with it. Prefix caching doesn’t help, because every query pulls a different set of documents — there is no shared prefix to cache.

The last two years produced a clean workaround, which I’ll call the chunk-cache family: TurboRAG, CacheBlend, Block-Attention, EPIC and friends. The trick is to encode each corpus chunk once, offline, in isolation, store its KV tensors, and at query time just concatenate the stored KVs instead of running a forward pass over thousands of tokens. Because the chunks never attended to each other during offline encoding, the spliced cache is an approximation of the true one, so each method adds a repair step: recompute the tokens whose KV deviates most (CacheBlend), re-apply positions and fine-tune the model to tolerate block-independent encoding (Block-Attention), etc.

That family solves cost per token of context. It does nothing about how many tokens of context you need. You still pay for the entire chunk — boilerplate, tangents, the two paragraphs that happened to sit next to the useful sentence. In multi-hop QA the actual evidence is often a handful of sentences scattered across several documents; everything else is redundancy and distractor noise, which hurts twice: it costs latency, and it degrades answers.

CoinRAG’s bet is that granularity is the unexploited axis. Not “cache better,” but “cache the same thing and consume less of it.”

[Problem] retrieved context is long => prefill latency dominates RAG serving
     |
     v
[Prior fix] chunk-level KV reuse: TurboRAG / CacheBlend / Block-Attention
     |       precompute per-chunk KV offline, splice + repair at query time
     |
     +--> unresolved: you still consume WHOLE chunks
     |    redundancy and distractors ride along for free (but not for cheap)
     |
     v
[Assumption] (a) the evidence is a few small spans inside those chunks
             (b) a span's KV *sliced out of its own chunk cache* already
                 carries the chunk's local context, unlike a span encoded alone
     |
     v
[Method] CoinRAG = two-stage retrieval (chunk -> nugget)
                 + KV slicing from offline chunk caches
                 + chunk-level context anchor
                 + learned adaptation to the assembled cache
     |
     v
[Evidence] LongBench multi-hop QA: dominates baselines on the
           latency-vs-F1 curve; +5.3% relative F1 at a fixed
           "fast prefill" budget
     |
     v
[Conclusion] under a latency budget, selecting finer units beats
             encoding coarse ones more cleverly

The Increment

One sentence: Before, the reusable unit of KV cache was the retrieval chunk; after, it’s a sub-chunk “nugget” that can be cut out of a chunk’s precomputed cache and recombined with others, so the model reads a short, dense context instead of a long, noisy one at the same or lower prefill cost.

Core Mechanism

Offline, CoinRAG does what TurboRAG does: run each chunk through the model and store its KV. The addition is that it also segments each chunk into information nuggets — small semantic units, roughly sentence- or proposition-sized — records their token spans, and indexes them for retrieval. Crucially the KV is not recomputed per nugget. A nugget’s cache is a row-slice of its parent chunk’s cache. That’s where “contextualized” comes from: the sentence was encoded while it could attend to its neighbors, so the slice inherits that local context for free. Encoding the nugget standalone would strip exactly the coreference and topical grounding that makes a fragment interpretable.

Online, retrieval happens twice. Stage one is ordinary chunk retrieval. Stage two runs inside the surviving chunks and scores nuggets against the query, keeping only the relevant ones. Then the assembly step gathers the corresponding KV slices — memory reads, no forward pass — and concatenates them into a synthetic context, together with a chunk-level context representation that acts as an anchor so a bag of fragments still reads as coherent evidence rather than a shuffled quote list. Positions are re-indexed (RoPE re-applied at the new offsets), and a small learned/recompute pass heals the seams, which is necessary because the model never saw this particular ordering of spans during pretraining. The abstract’s phrase “learned contextual representation” implies training is involved, in the same spirit as Block-Attention’s fine-tuning; from the abstract alone I can’t tell whether it’s a fine-tune of the base model, a light adapter, or a learned selection policy.

The payoff compounds. A shorter assembled sequence means less repair compute during prefill, fewer KV rows to attend over during decode, and less distractor content for the model to be misled by. That’s why the claim is a Pareto shift rather than a pure speedup: quality goes up while latency goes down.

OFFLINE  (once per corpus)
  chunk C --> [ LLM forward ] --> KV(C) stored on disk / GPU-adjacent tier
                                  span table: n1 n2 n3 ... nk
                                  nugget embeddings --> stage-2 index

ONLINE   (per query q)

  q --> stage 1: chunk retriever --> C1  C2  C3
  q --> stage 2: nugget retriever, scoped inside C1..C3
                        |
                        v
             selected spans:  C1[n2]  C2[n1]  C2[n5]  C3[n3]
                        |
                        v
             KV SLICING (no forward pass, just reads)
             KV(C1)[rows of n2] , KV(C2)[rows of n1], ...
                        |
                        v
             ASSEMBLE
             [ sys ][ chunk-level anchor ][ n2 ][ n1 ][ n5 ][ n3 ][ q ]
               + position re-index (RoPE at new offsets)
               + small learned / recompute pass to heal seams
                        |
                        v
             DECODE over a SHORT, dense sequence
             (vs. baseline: decode over 4x longer spliced chunks)

Think of it as a film editing room. Shooting is expensive, so you shoot each scene once — that’s the offline chunk prefill, and the exposed footage is the stored KV. When a query arrives you are not re-shooting anything; you are cutting. Stage-one retrieval picks which scene reels to pull from the shelf; stage-two retrieval picks which shots inside those reels actually belong in this cut. The key editorial discipline is that you cut the shot out of the developed reel rather than re-staging the actor on a bare set: the clip keeps the lighting and the set dressing of the original scene, which is exactly what a KV slice keeps and what an independently-encoded fragment loses. The chunk-level anchor is the establishing shot you leave in so the audience knows where they are. Position re-indexing is renumbering the timecodes after the splice. And the learned repair pass is the continuity-and-color-grading pass that keeps the cuts from feeling jarring. The output is a ninety-second trailer that answers the question, not the four-hour dailies.

Key Concepts

• KV cache reuse, and why it’s an approximation: When a transformer reads text, each token computes a key and a value vector at every layer, and later tokens attend to them. Prefill is precisely the cost of computing those vectors for the whole context. If a document is going to appear in many queries, you could compute its keys/values once and store them. The catch: attention is contextual. A key computed while the model was looking at only document A is *not the key it would have computed if documents B and C had been in front of it too. So splicing independently-computed caches gives you a context the model never actually would have produced — usually close enough, sometimes not, which is why every method in this family bolts on a repair step.

• Information nugget: A term borrowed from IR evaluation, where a “nugget” is a minimal atomic fact that an answer must contain — TREC-style nuggetized scoring works this way. Concretely, for “which university employed the physicist who proposed the wave equation?”, the nuggets are “Schrödinger proposed the wave equation” and “Schrödinger held a chair at the University of Zurich” — two sentences, possibly from two documents, buried in maybe 4,000 tokens of retrieved biography. Nuggets are the unit of *evidence; chunks are the unit of storage. CoinRAG’s move is to stop conflating them.

• Contextualized slicing vs. isolated encoding: Take the sentence “He was appointed there in 1921.” Encoded on its own, it is nearly useless — “he” and “there” resolve to nothing. Encoded as part of its chunk and then sliced out, its key/value vectors were already computed with the antecedents visible, so the resolved meaning is baked into the numbers even though the antecedent tokens are no longer in the sequence. This is the paper’s central lever, and also its main fragility: the resolution lives only in the vectors, and how well it survives re-positioning and truncation of the surrounding rows is an empirical question.

Framework Shift

Before (chunk-cache family)              After (CoinRAG)
                                          
  q                                        q
  |                                        |
  v                                        v
retrieve top-k chunks                    retrieve top-k chunks
  |                                        |
  v                                        v
+----------+----------+----------+       score nuggets inside them
|  KV(C1)  |  KV(C2)  |  KV(C3)  |         |
+----------+----------+----------+         v
  ^ reuse whole blocks              +--+  +-+  +---+  +-+
  |                                 |n2|  |n1| |n5 |  |n3|   slices
  |  ~4000 tokens                   +--+  +-+  +---+  +-+
  |  of which ~10% is evidence        |  ~600 tokens, mostly evidence
  v                                    v
repair seams --> decode            anchor + slices --> repair --> decode
                                          
cost: f(size of chunks)            cost: f(size of evidence)
noise: fully inherited             noise: filtered at stage 2

One sentence: from making long context cheap to encode to making the encoded context short, the core shift is decoupling the unit of caching from the unit of consumption.

Expert Assessment

Problem choice: Real gap, and well-positioned. The chunk-cache line of work and the context-compression line of work (LLMLingua, RECOMP, Provence, sentence-level rerankers) have been running in parallel with almost no contact, because compression normally forces you to re-encode the compressed text — which throws away your cache. Noticing that you can compress *in cache space, by slicing rather than re-encoding, is the kind of connection that should have been made a year ago. It sits squarely on the field’s trajectory toward treating the KV cache as a first-class, addressable data structure rather than a byproduct of a forward pass.

Method maturity: Clever, not brute force — the “slice, don’t re-encode” insight is the whole paper and it’s a good one. Two things temper my enthusiasm. First, the system is now three coupled components (chunk retriever, nugget retriever, learned repair), and papers with three knobs usually hide an ablation showing one knob does most of the work; I’d want to know whether an oracle-free stage-two retriever is really the source of the gain or whether shorter contexts alone explain it. Second, a genuinely simpler baseline exists and is easy to overlook: retrieve nuggets, then just *prefill the concatenated nugget text normally. If the context is now only ~600 tokens, plain prefill is already fast, and the entire cache-slicing apparatus may be buying less than it appears. The paper’s advantage over that baseline has to come from either latency at scale or from contextualization that raw re-encoded fragments lack — and that comparison is the one I’d read first.

Experimental integrity: I’m working from the abstract, so this is provisional. Two flags. (1) The headline is “+5.3% relative F1 under a fast prefill latency budget.” Relative F1 on LongBench multi-hop tasks means roughly 1.5–2.5 absolute points, and those datasets are noisy enough that the result needs multiple seeds and per-dataset breakdowns to be convincing. Reporting relative gains under a *chosen budget also invites cherry-picking the budget; the Pareto plot mitigates this if the curves are dense, but only if the crossing points are shown honestly. (2) The accounting boundary matters enormously. Stage-two nugget retrieval is real online work. If it’s excluded from “prefill latency” while the baselines’ entire cost sits inside prefill, the comparison flatters CoinRAG. Also worth checking: whether the learned repair component means CoinRAG got a fine-tune that CacheBlend didn’t, and what the nugget index costs in memory relative to the chunk index. Evaluating only multi-hop QA is a narrow slice — fragment-based context should be much riskier for summarization or any task needing discourse-level coherence, and silence on that is conspicuous.

Writing quality: The abstract spends a sentence and a half on the coin metaphor, which is a bad trade in a venue where reviewers are counting contributions. “Seamlessly assembles” is doing a lot of unexamined work — the seams are the hard part and the honest version of that sentence names the mechanism. The section I’d most want rewritten is the cost model: a clean, explicit budget (offline compute, storage, index size, stage-two retrieval latency, repair FLOPs, decode length) would convert this from “nice numbers” to “deployable,” and it’s exactly what an infrastructure-minded reader will look for and, I suspect, not find.

Verdict: weak accept — a genuinely good idea (compress inside cache space instead of text space) with a plausible efficiency story, held back by a narrow evaluation and an accounting boundary that needs to be shown, not asserted.

Takeaways

• The transferable primitive: slice, don’t re-encode. Any pipeline that caches representations of text can adopt this. If you already have a precomputed representation of a whole document, extracting a sub-span from that representation preserves context that re-encoding the sub-span destroys. This applies to late-interaction retrievers, document rerankers, and any embedding cache with span-level addressability — not just KV caches.

• Decouple the storage unit from the consumption unit. Chunk size in RAG is usually one number serving three different jobs: retrieval granularity, caching granularity, and context granularity. They have different optima. Splitting them is a cheap architectural win independent of anything else in this paper.

• Hierarchical retrieval as a latency instrument. Two-stage retrieval is usually framed as a quality trick (recall then precision). Here it’s a *cost trick: stage two exists to shrink what the LLM reads. If you’re latency-bound, a cheap in-chunk filter may buy more than a better first-stage retriever.

• The “anchor” pattern. Keeping one coarse-grained context representation alongside fine-grained fragments is a reusable hedge against over-aggressive pruning. Worth borrowing whenever you compress context and worry about losing the thread.

• What I would not take on faith: that fragment assembly is safe outside short-answer multi-hop QA. Before shipping this for anything generative, test on tasks where the model must respect discourse order.