Context-Driven Incremental Compression for Multi-Turn Dialogue Generation

Paper: 2606.12411
Authors: Yeongseo Jung, Jaehyeok Kim, Eunseo Jung, Jiachuan Wang, Yongqi Zhang, Ka Chun Cheung, Simon See, Lei Chen
Categories: cs.CL, cs.LG

The Gap

Existing dialogue agents either compute attention over the entire growing history (O(n²)), or apply naive truncation/summarization at each turn that discards information permanently. Methods like Memformer or Compressive Transformer do compress context, but they treat the compressed state as a fixed, immutable blob — they never revise stale memories when later turns reveal contradictions or disambiguation. This means errors compound over long dialogues. The paper empirically demonstrates this fragility: even a strong compressor like RecurrentGPT degrades after ~30 turns on multi-benchmark evaluations.

What’s missing is a compression scheme that (a) treats a dialogue as multiple interleaved topical threads, (b) allows each thread’s compressed representation to be read, revised, and written back at each turn, and (c) maintains a single compact memory rather than unbounded history. This paper fills exactly that gap.

        Problem: long dialogue -> redundant attention + error compounding
                |
                v
        Assumption: cross-turn memory sharing + revision are necessary
                |
                v
        Method: C-DIC -> per-thread revisable compression + TBPTT
                |
                v
        Evidence: stable PPL & latency over 300 turns on 4 benchmarks
                |
                v
        Conclusion: incremental, thread-aware compression is scalable

The Increment

One sentence: Before this paper, long-dialogue models either truncated history (losing fidelity) or used fixed compression (error accumulation); after this paper, we have a method that incrementally per-thread compresses and revises compressed memories, achieving consistent performance regardless of turn count.

Core Mechanism

C-DIC consists of three main components: a Dialogue Memory (a single compact vector set), a Thread Decoder that identifies which threads are active in the current turn, and a Retrieve–Revise–Write-back loop that operates each turn.

At each turn, the model first encodes the latest utterance and retrieves from the dialogue memory the compressed representation of the thread(s) most likely related to this utterance (using attention over thread IDs). Then it revises those retrieved representations: the new utterance may add, correct, or eliminate facts in the thread representation (e.g., “actually, I prefer the blue one, not red” updates the color slot). The revised representation is written back into the same memory slot, overwriting the stale version. Meanwhile, threads not involved remain untouched. This loop is lightweight because memory size is bounded (e.g., 32 threads, each a single hidden vector), so retrieval and revision are cheap.

Finally, the model generates the next response conditioned on the updated dialogue memory (plus the current utterance). To learn cross-turn dependencies without backpropagating through the entire history, the authors adapt TBPTT: they unroll the retrieval–revision–generation loop for a fixed window (say 8 turns), backpropagate through that window, and truncate gradients beyond. This makes training feasible on long sequences.

                        current utterance
                              |
                              v
      [Dialogue Memory] <--- Retrieval (inner product with thread IDs)
          |                        |
          |                        v
          |                  Revised representation (add/update facts)
          |                        |
          +-----------> Write-back (overwrite old slot)
                              |
                              v
                      Generate next response
                              |
                              v
                         (next turn)

Structural metaphor: Imagine a group of researchers writing a shared wiki page about a conversation. Each topical thread (e.g., “movie preferences”, “travel plans”, “family details”) gets its own wiki entry. At each new message, a researcher reads the relevant entries, edits them (revises outdated info, adds new facts), and saves the updated entries. The wiki stays compact because old, inactive threads are not expanded — only the active ones are touched. The whole process takes a fixed number of edits per turn, regardless of how many total messages have accumulated. Similarly, C-DIC keeps a fixed-size “wiki” of compressed thread memories and performs incremental edits per turn.

Key Concepts

• Incremental Compression: Unlike batch compression (re-encode the entire history each turn), incremental compression updates only the parts of the compressed state that are relevant to the new turn. This is like Git: you don’t rewrite the whole repo every commit; you only commit the diff. C-DIC treats each thread as a file, and each turn as a small patch to that file. This saves compute and prevents forgetting previous threads that are not touched.

• Revisable Memory State: A fixed compressed memory that can be overwritten. This is different from append-only memory (e.g., RNN hidden state that grows or gets overwritten only by concatenation). The key is that the revision operation can subtract or correct information — a negation sentence (“no, I meant the red car”) can remove a fact that was previously stored. In practice, the revise step is a learned gated operation similar to a GRU cell, but operating on the compressed vector.

• Thread-Based Partitioning: The dialogue is automatically split into multiple threads (learned unsupervised). Each thread corresponds to a coherent subtopic. The model uses attention to assign each new utterance to one or more threads. This prevents interference between unrelated topics: updating the “movie thread” does not corrupt the “travel thread”. Without thread partitioning, a single compressed vector would be forced to store everything, causing catastrophic forgetting when new turns overwrite unrelated information.

Framework Shift

Before (mainstream approach):  
                                                                  
  [Entire history] -> [Fixed compressor] -> [Fixed compressed vector] -> Generate  
       (every turn re-encode all tokens)        (never revised)          (error accumulates)  

After (this paper):  
                                                                  
  [Current utterance] + [Thread-aware retrieval] -> [Revise thread(s)] -> [Write-back]  
                                                                                   |  
                                                                                   v  
                                                                        [Compact memory] -> Generate  
                                                                        (fixed size, per-thread)  

One sentence: From monolithic irreversible compression to threaded incremental revision, the core shift is making the compressed state writable and thread-aware, so that each turn only modifies relevant parts and consolidates information across time.

Expert Assessment

Problem choice: Genuine gap. The growth of chat history is a bottleneck for all production dialogue systems; simple truncation is already inadequate for tasks requiring long context (e.g., customer-support diagnosis, role-playing games). The paper targets a real pain point.

Method maturity: Clever but not surprising. The retrieval-revision loop is essentially an attention-gated update inspired by Neural Turing Machines and Differentiable Neural Computers. The main novelty is applying it to multi-thread compression in dialogue. A simpler approach could be to just use sliding-window attention on full text (Longformer-style) — the paper shows their method is more compute-efficient (stable latency < 50ms per turn vs O(n) for long windows). Fair.

Experimental integrity: The baselines are fair (truncation, naive summarization, Memformer, RecurrentGPT, TwoMe) and the results are convincing: C-DIC achieves the best perplexity on four benchmarks (Persona-Chat, Wizard-of-Wikipedia, Topical-Chat, Multi-Session Chat) and maintains stable latency across 300 turns. One red flag: the paper only reports perplexity, not human evaluation or downstream task success (e.g., response relevance, fact accuracy). Perplexity can be gamed.

Writing quality: The abstract is clear, but Section 4 (experiments) buries important implementation details (e.g., number of threads, memory dimension, TBPTT window size) in the appendix. The main text should at least mention these hyperparameters. Also, the metaphor is not developed enough in the paper itself — they should have connected it to the “wiki” analogy for better intuition.

Verdict: weak accept — solid engineering contribution with clear practical impact, but lacks rigorous human evaluation and would benefit from analysis of failure cases (e.g., what happens when thread collapse occurs?).

Takeaways

• Revisable memory is a viable pattern for long-sequence modeling: if you can define a set of “slots” that correspond to latent topics, you can update them with new information and discard old. This generalizes to video description (update scene representations), document processing (keep a compressed summary per section), and continual learning.

• Thread retrieval via inner-product attention is cheap and surprisingly effective: the paper uses a simple dot-product against learned thread embeddings to assign utterances to threads. No complex clustering — it just works. Practitioners can copy this design for any task requiring topic-level segmentation.

• TBPTT window length matters: they used a fixed window of 8 turns. This is a hyperparameter that can be tuned per domain. Shorter windows train faster but may miss long-range dependencies; longer windows capture more but risk gradient explosion. The ablation study (Appendix) shows diminishing returns after 16 turns.

• If you build a production chatbot, consider C-DIC’s memory design: your token budget stays constant (e.g., 1024 tokens for the entire conversation, thanks to compression). This avoids the ever-growing context problem and lets you serve long dialogues with predictable cost.