LatComm: Reusable Context Compression for Latent Communication between Large Language Models

Paper: 2610.26840
Authors: Huawei Lin, Jinting Liu, Tan Yu, Weijie Zhao
Categories: cs.AI, cs.CL

The Gap

In compound AI systems, collaborative multi-agent swarms, and enterprise retrieval pipelines, a single massive context—such as an extensive API specification, a medical case file, a legal contract, or a full code repository—frequently serves as the shared reference for dozens or hundreds of downstream questions.

Under current serving paradigms, communication between disparate LLMs relies on natural language text transmission: the sender dumps raw text into the receiver’s prompt. This introduces a devastating computational redundancy:

• Redundant Prefill Overhead: The receiver LLM must process and re-encode the entire multi-thousand-token document from scratch for every single question, consuming vast GPU FLOPs and saturating KV cache memory.
• The Failure of Existing Compression: Prior latent communication and context compression approaches suffer from crippling structural limits:
  1. Single-Vector Pooling (e.g., dense embeddings): Destroys granular token-level evidence, collapsing accuracy on detailed reading comprehension.
  2. Query-Dependent Compression (e.g., soft prompting / TTT): Compresses the context specifically for one query, destroying reusability across future questions.
  3. Homogeneous Weight Binding: Requires the sender and receiver to be identical model checkpoints sharing the exact same vocabulary and hidden dimensions.

   THE REDUNDANT TEXT PREFILL TRAP IN MULTI-AGENT PIPELINES

   100k-Token Long Legal Contract
          |
          +-----------------------+-----------------------+
          |                       |                       |
          v                       v                       v
   [Question 1]            [Question 2]            [Question 50]
   "Liability clause?"     "Governing law?"        "Termination dates?"
          |                       |                       |
          v                       v                       v
   +-----------------------------------------------------------+
   | Receiver LLM must run full forward prefill pass over     |
   | the SAME 100k tokens 50 CONSECUTIVE TIMES!                |
   | Total Tokens Processed: 5,000,000 tokens!                 |
   | VRAM Choked; Prefill Latency Dominates Serving Costs.     |
   +-----------------------------------------------------------+
          |
          v
   LatComm SOLUTION: ONE ENCODING, UNLIMITED REUSABLE READS
   - Frozen Encoder compresses 100k tokens ONCE into a fixed Latent Packet
   - Frozen Receiver reads relevant packet slices via Prompt-Gated Cross-Attention
   - Receiver input slashed 57x; Multi-query answering runs 3x faster!

The Increment

One sentence: LatComm establishes an asynchronous latent communication interface that encodes long textual contexts once into a compact, query-agnostic neural packet, allowing a separate frozen receiver LLM to selectively query the packet via prompt-gated cross-attention—reducing receiver input tokens by 57x and achieving 3x faster end-to-end question answering.

Core Mechanism

LatComm decouples context digestion into asymmetric, one-time encoding and query-conditioned selective reading:

1. One-Time Query-Agnostic Packetization:
   • A frozen sender LLM processes the long context document without access to downstream questions.
   • Intermediate activations across landmark positions are downsampled and projected into a fixed-size Latent Packet $\mathcal{P} \in \mathbb{R}^{M \times d}$ (where $M \ll N$, typically achieving a $16\times$ to $64\times$ token reduction).
   • This packet is cached in GPU memory or transmitted across nodes as a lightweight tensor payload.
2. Prompt-Gated Cross-Attention Reading:
   • The receiver LLM can be a completely different frozen model architecture.
   • When a specific question $\mathbf{q}$ arrives, the receiver generates query vectors that attend to the latent packet through lightweight, trainable cross-attention bridge layers.
   • An in-situ gating module $\mathbf{g}(\mathbf{q})$ computes prompt-dependent attention masks, allowing the receiver to selectively extract needle-level evidence from the packet while ignoring irrelevant background regions.
3. Completely Frozen Foundation Models:
   • Neither the sender LLM nor the receiver LLM undergoes weight updates.
   • Only the compact projection adapters and cross-attention gating parameters are optimized, preserving the broad reasoning priors and conversational alignment of both models.
4. Spectacular Throughput and Accuracy:
   • Evaluated across 12 demanding multiple-choice QA benchmarks.
   • Outperforms conventional dense single-vector baselines by up to 10.2 percentage points.
   • On the QuALITY long-context benchmark, LatComm cuts receiver input tokens by $57\times$, speeding up document-level question answering by $3\times$ in wall-clock time.

   LatComm LATENT COMMUNICATION ARCHITECTURE

   Long Background Document Context (N Tokens)
                    |
                    v
   [Frozen Sender LLM] (Runs ONCE per document)
                    |
                    v
   [Compressor Projection Module]
                    |
                    v
   ===>> CACHED LATENT PACKET P (M Tokens, M = N / 57) <<===
                    |
         +----------+----------+
         |                     |
         v (Question 1)        v (Question 2)
   "What is statute X?"  "What are damages in clause Y?"
         |                     |
         +----------+----------+
                    |
                    v
   [Prompt-Gated Cross-Attention Bridge]
   (Dynamically extracts relevant packet slices based on query prompt)
                    |
                    v
   [Frozen Receiver LLM]
                    |
                    v
   Fast Sub-Second Answers (3x Faster Wall-Clock Serving!)

Key Concepts

• Latent Communication: Passing internal continuous activation tensors between distinct neural models rather than translating into discrete natural language text tokens.
• Query-Agnostic Context Packet: A dense neural representation of a document that retains sufficient entropy to answer arbitrary future questions without being conditioned on a specific query prompt.
• Prompt-Gated Reading: An attention mechanism where the incoming question modulates cross-attention weights over pre-computed latent memory slots.

Framework Shift

Before (Text-Based Context Transmission in Multi-Agent Loops):
  Protocol: Serialize full context as text; dump into every downstream agent's prompt.
  Computation: N tokens re-encoded across Q questions = N * Q prefill complexity.
  Cost: Enormous redundant compute; saturates memory bandwidth and blows up serving costs.

After (LatComm Reusable Latent Interface):
  Protocol: Encode context once into a compact latent packet P; query P via gated cross-attention.
  Computation: N tokens encoded ONCE + (N / 57) tokens read across Q questions.
  Efficiency: 57x fewer receiver input tokens; 3x faster wall-clock serving; zero loss in QA fidelity.

Expert Assessment

Problem choice: Immediate architectural relevance for multi-agent systems and enterprise knowledge retrieval. Re-encoding identical long contexts for every user query is one of the most egregious compute wastes in modern cloud AI.

Method maturity: Decoupling the encoder from the reader while keeping both foundational models frozen is a masterclass in practical systems design, avoiding the need for multi-billion parameter end-to-end retraining.

Experimental integrity: Validated across 12 diverse question-answering benchmarks, complete with rigorous ablations comparing single-vector, pooling, and multi-vector cross-attention topologies.

Writing quality: Highly lucid, with clear systems-level diagrams and compelling latency profiling on long-context datasets.

Verdict: strong accept — A groundbreaking systems paper that provides the blueprints for scalable latent communication in compound AI architectures.

Takeaways

• Stop re-encoding identical static contexts as raw text for every query; encode the document once into a reusable latent packet.
• Keep sender and receiver foundation models frozen and use lightweight prompt-gated cross-attention to bridge latent representations.
• In multi-agent pipelines and enterprise document chat, deploy latent packets to achieve up to 57x input compression and 3x end-to-end speedups.