ACCT: Asymmetric Cross-LLM Context Transfer

Paper: 2610.42116
Authors: Yuzhe Liu, Haolin Tian, Tonghan Wang (Tsinghua University)
Categories: cs.CL, cs.AI, cs.LG

The Gap

Production large language model (LLM) infrastructures are rapidly moving away from monolithic single-model endpoints toward heterogeneous serving architectures (e.g., speculative decoding, cascade routing, multi-agent systems, and collaborative edge-cloud pipelines). In these systems, small models (e.g., 3B–7B) perform initial triage, routing, or verification, while larger models (e.g., 8B–70B) handle complex reasoning.

However, existing serving engines suffer from a massive efficiency barrier: the context re-computation bottleneck.

1. Redundant Prefill at Massive Context Lengths:
   • Modern interactions feature long conversational histories, codebases, or multi-document retrieval contexts spanning 10K to 32K+ tokens.
   • When a request is escalated from a small model to a large model, the target model cannot read the source model’s Key-Value (KV) cache due to differences in hidden dimensions, head counts, and layer depths.
   • Consequently, the target model must re-encode the entire context from raw text, incurring substantial prefill FLOPs and exploding Time-To-First-Token (TTFT) latency.
2. The Failure of Naive KV Sharing:
   • Direct dimension projection fails because different layers and attention heads across model scales capture drastically different semantic abstractions.
   • Simply pruning the context with uniform heuristics degrades multi-hop reasoning and domain QA.

               THE HETEROGENEOUS SERVING BOTTLENECK

  Traditional Setup:
  [User Prompt: 20K Tokens]
          │
          ├──> Small Model (4B): Computes KV Cache (20K tokens) ──> Routes to 8B
          │
          └──> Target Model (8B): MUST RECOMPUTE 20K Tokens Prefill from scratch!
               * Wastes 95% of prefill compute
               * Adds seconds of Time-To-First-Token (TTFT) latency
               * VRAM exhaustion under concurrent long contexts

---

The Increment

One sentence: ACCT (Asymmetric Cross-LLM Context Transfer) decouples context computation from downstream inference by compiling precomputed source-model KV caches into sparse, query-relevant target-model KV states—cutting KV cache footprint by 75% while achieving 6.5× to 21.2× speedups over native target prefill at ~19K tokens.

                              ACCT ARCHITECTURE

  Source Model KV Cache (Layer l_s, Head h_s) + Query q
                         │
                         ▼
             [1. The Cartographer]
     Ranks content tokens for each target layer
     and KV head based on query-relevance
                         │
                         ▼
              [2. The Allocator]
     Dynamically allocates token budgets across
     target layers/heads under a global budget (e.g., 25%)
                         │
                         ▼
              [3. The Compiler]
     Linearly projects selected source KV vectors
     into target-compatible Key-Value representations
                         │
                         ▼
             [Target Model Inference]
     Directly mounts transferred sparse KV cache!
     Skips prompt prefill -> generates response immediately!

---

The Anatomy

1. Three Lightweight Components

ACCT bridges within-family model pairs (e.g., Qwen3 4B $\leftrightarrow$ 8B, LLaMA 3B $\leftrightarrow$ 8B, Qwen2.5 7B $\leftrightarrow$ 32B) through three coordinated modules:

A. The Cartographer: Query-Grounded Token Selection

For each target layer $l_t$ and attention head $h_t$, the Cartographer ranks all context tokens according to their predicted relevance to the current query: $S_{l_t, h_t} = \text{Score}\left(K_s, V_s, q_s; l_t, h_t\right)$ Instead of computing full attention over all layers, it leverages query representations from the source model to prioritize the critical tokens required for target-model reasoning.

B. The Allocator: Dynamic Cross-Layer KV Budgeting

Not all layers or attention heads require equal context depth. Some heads track broad syntactic boundaries with high token sparsity, while others maintain precise multi-hop factual linkages. Under a total global budget $B$ (e.g., 25% of full context), the Allocator assigns head-specific capacities $b_{l_t, h_t}$ such that: $\sum_{l_t} \sum_{h_t} b_{l_t, h_t} \le B$ This concentrates KV capacity on the most information-dense attention heads.

C. The Compiler: Cross-Model Feature Projection

For the retained tokens, the Compiler projects source KV vectors into target KV dimensions using learned linear mappings: $K_t^{(l_t, h_t)} = W_K \cdot K_s^{(l_s, h_s)}, \quad V_t^{(l_t, h_t)} = W_V \cdot V_s^{(l_s, h_s)}$ The target model mounts these compiled tensors directly into its KV cache buffer, bypassing token embedding and prefill transformer blocks entirely.

2. Computational Latency Profile

At inference time, the target model’s prefill phase is replaced by the lightweight conversion process: $T_{\text{ACCT}} = T_{\text{source prefill}} + T_{\text{conversion}}$ When the source model has already prefilled the context (as in multi-turn dialogues, agentic memory lookups, or routing gates), the target model experiences zero context prefill overhead, needing only to encode the brief user query.

---

Quantitative Evaluation

The authors evaluate ACCT across three major open-source model families (Qwen3, LLaMA, Qwen2.5) across 6 transfer directions spanning 5 general-capability benchmarks and 6 context-dependent reasoning tasks.

1. Task Performance at 4× Compression (75% Cache Reduction)

Model Direction	Method	MMLU (Acc)	PIQA (Acc)	SearchQA (F1)	TriviaQA (F1)	BoolQ (Acc)	PubMedQA (Acc)
Qwen3 4B → 8B	Target Full (100% KV)	59.70%	73.20%	46.65%	50.48%	86.30%	48.80%
	ACCT (25% KV)	65.70%	73.10%	67.07%	66.31%	61.90%	40.20%
	Relative Performance	1.10×	1.00×	1.44×	1.31×	0.72×	0.82×
Qwen3 8B → 4B	Target Full (100% KV)	63.60%	69.20%	38.56%	37.68%	84.00%	34.00%
	ACCT (25% KV)	58.20%	69.00%	50.33%	54.51%	60.20%	49.60%
	Relative Performance	0.92×	1.00×	1.31×	1.45×	0.72×	1.46×
Llama 3B → 8B	Target Full (100% KV)	58.60%	77.40%	63.44%	78.71%	82.00%	70.00%
	ACCT (25% KV)	41.90%	69.30%	66.88%	69.71%	63.50%	53.60%
	Relative Performance	0.72×	0.90×	1.05×	0.89×	0.77×	0.77×
Qwen2.5 7B → 32B	Target Full (100% KV)	72.40%	80.30%	44.05%	34.10%	85.30%	57.00%
	ACCT (25% KV)	64.60%	70.20%	46.47%	33.58%	56.60%	47.00%
	Relative Performance	0.89×	0.87×	1.05×	0.98×	0.66×	0.82×

Key Result: Even under a strict 4× compression ratio (75% cache reduction), ACCT actually outperforms Target Full on open-domain knowledge retrieval tasks (+20.4 F1 points on SearchQA and +15.8 F1 points on TriviaQA for Qwen3 4B $\rightarrow$ 8B) because the Cartographer filters out distracting context noise.

2. Context Prefill Latency Acceleration

Across all six transfer directions at ~19K tokens context length:

• Prefill Latency Speedup: ACCT achieves 6.5× to 21.2× speedup over native target context prefilling.
• Computation Time Reduction: Slashes prefilling time by up to 95.3%.
• Memory Footprint: Reduces global KV cache storage by 75% at $4\times$ compression and 50% at $2\times$ compression.

---

Key Takeaways

1. Context is Reusable Across Models: Model architectures within the same family learn sufficiently compatible semantic spaces to allow direct cross-scale KV cache translation.
2. Filtering Out Noise Beats Full Attention: Retaining only 25% of query-relevant KV tokens often outperforms 100% full context retention on QA tasks, demonstrating that selective sparse caching acts as an implicit denoising mechanism.
3. Game Changer for Cascade Routing & Multi-Agent Runtimes: ACCT removes the prohibitive prefill latency penalty when escalating queries from small router models to giant reasoning models.

                    异构大模型部署中的算力与延迟瓶颈

  传统流程：
  [用户长上下文输入: 20K Tokens]
          │
          ├──> 源小模型 (4B): 耗时计算 20K Tokens 的 KV Cache ──> 判定需要转交 8B 模型
          │
          └──> 目标大模型 (8B): 必须从零重新进行 20K Tokens 的 Prefill 计算！
               * 浪费了超过 95% 的前置计算算力
               * 产生数秒的首字生成延迟（TTFT 严重超时）
               * 多并发场景下显存迅速爆满崩溃

                              ACCT 系统技术架构

  源模型预计算的 KV Cache (层 l_s, 头 h_s) + 用户提问 Query q
                         │
                         ▼
             [1. 制图师 Cartographer]
     基于源端 Query 表征，对目标模型各层与各头
     的 Context Token 进行相关性打分与排序
                         │
                         ▼
              [2. 分配器 Allocator]
     在全局总预算（如仅保留 25% KV）约束下，
     对各层各头动态分配非对称的 Token 保留容量
                         │
                         ▼
              [3. 编译器 Compiler]
     利用轻量线性变换，将选中的源 KV 向量直接
     投影编译为目标模型兼容的 Key-Value 张量
                         │
                         ▼
             [目标大模型直接挂载推理]
     直接挂载编译后的稀疏 KV Cache，完全跳过 Prefill 阶段！
     仅需编码用户单句提问，立即可开始流式生成！