Visual Token Routing: Learning to Keep Tokens That Matter for Computer-Use Agents

Paper: 2611.06854
Authors: Pranjal Aggarwal, Sean Welleck (Carnegie Mellon University)
Categories: cs.CV, cs.AI, cs.LG

The Gap

Computer-Use Agents (CUAs)—autonomous models that operate operating systems and desktop software by interpreting screenshots and executing mouse clicks and keystrokes—have demonstrated breakthrough capabilities on environments like OSWorld and CUA-World.

However, deploying these agents in real-world workflows and real-time environments (such as interactive software and video games) is severely bottlenecked by a fundamental scalability crisis: the visual token explosion.

1. Massive Context Inflation: Foundation vision-language models encode high-resolution desktop displays into enormous token sequences. For example, a single 1080p ($1920 \times 1080$) display consumes ~2,040 visual tokens. In a standard 100-step desktop workflow, maintaining an uncompressed visual history forces the agent context beyond 200,000 tokens by task completion.
2. Extreme Semantic & Temporal Redundancy: In desktop computing, over 95% of screen pixels remain identical from one step to the next (e.g., the desktop background, taskbar, browser navigation chrome, static document margins). Even when pixels do change—such as an auto-playing video ad in a background tab—those changes are completely irrelevant to typing a filename in a save dialog.
3. Flaws in Naive Reduction Baselines:
   • Lowering Global Resolution: Shrinking images to 50% or 25% degrades fine-grained UI details, rendering small icons, single spreadsheet cells, and 10pt text unreadable.
   • Heuristic Pixel-Diff & Video Codecs (H.264): Rule-based change detection tracks any pixel delta blindly. If a banner ad flashes in the corner, pixel-diff retains hundreds of useless tokens while failing to keep static context that the agent must read.

                   THE CUA VISUAL TOKEN BOTTLENECK

  Step 1               Step 25             Step 50              Step 100
  [Screenshot: 2K] ──> [History: 50K] ──> [History: 100K] ──> [History: 200K+ Tokens!]
         │                   │                   │                    │
  Prefill: 120ms       Prefill: 450ms      Prefill: 980ms       Prefill: 2,400ms (Unusable in RT!)
  Cost: $0.01          Cost: $0.15         Cost: $0.45          Cost: $1.80 per task

  Reality: 95% of pixels are identical (taskbars, margins) or irrelevant (background animations).

---

The Increment

One sentence: Carnegie Mellon researchers introduce Visual Token Routing (VTR), an end-to-end framework pairing vision-language agents with a lightweight spatio-temporal router that dynamically drops 2.5× to 5.8× of visual tokens outside key anchor frames—speeding up prefill by up to 2.9× and reducing inference cost by up to 3.3× with zero loss in action prediction accuracy.

                   VISUAL TOKEN ROUTING (VTR) PIPELINE

  Observations [ o_{t-3}, o_{t-2}, o_{t-1}, o_t ]
       │
       ├─ o_{t-3} (Anchor Frame): Preserved 100% in full resolution
       │
       └─ Subsequent Frames [ o_{t-2}, o_{t-1}, o_t ]:
              │
              ▼
       ┌─────────────────────────────────────────────────────────┐
       │   Spatio-Temporal Router f_phi                          │
       │   Examines temporal trajectory per spatial patch j      │
       │   Computes Bernoulli retention probability p_{t,j}      │
       └─────────────────────────────────────────────────────────┘
              │
              ├─ Static Taskbar / Margin ──> DROPPED (0 tokens)
              ├─ Irrelevant Video Ad     ──> DROPPED (0 tokens)
              └─ Typed Save Dialog Name  ──> KEPT (Full-res tokens with position embeddings)
              │
              ▼  (2.5x - 5.8x fewer visual tokens)
       ┌─────────────────────────────────────────────────────────┐
       │   Vision-Language Policy pi_theta (vLLM Engine)         │
       │   Jointly trained: Predicts actions from sparse tokens   │
       └─────────────────────────────────────────────────────────┘
              │
              ▼
       Precise Mouse & Keystroke Actions (Zero accuracy degradation!)

---

Core Framework & Mechanisms

1. Spatio-Temporal Patch Router

Rather than relying on hand-crafted heuristics, VTR integrates a lightweight neural router $f_\phi$:

• For each spatial patch region $j$, the router takes the temporal concatenation of visual features from the anchor up to the current frame: $[v_{\text{anchor}, j}; \dots; v_{t, j}]$.
• It outputs a scalar retention probability $p_{t, j} \in [0, 1]$.
• A discrete binary gate $g_{t, j} \sim \text{Bernoulli}(p_{t, j})$ determines whether patch $j$ enters the language model.
• Kept tokens maintain their original 2D positional embeddings, allowing the language backbone to reason spatially across the canvas.
• Anchor frames are periodically refreshed (every block of $B$ observations) to ensure that the agent never drifts out of sync with ambient desktop changes.

2. Dual Ascent Optimization for Adaptive Token Budgets

How can the model learn which tokens are safe to drop without manual threshold tuning? The authors frame training as a constrained optimization problem: $\min \mathcal{L}_{\text{tok}} \quad \text{subject to} \quad \bar{r} \le r^\star$ where:

• $\mathcal{L}_{\text{tok}}$ is the expected fraction of visual tokens retained outside anchor frames.
• $\bar{r}$ measures performance degradation relative to a full-history model, normalized by the difficulty of the action token: $i_n = \ell_n^{\text{blind}} - \ell_n^{\text{full}}, \quad d_n = \ell_n^{\text{sparse}} - \ell_n^{\text{full}}$ $\mathcal{L}_{\text{deg}} = \frac{\sum_n [i_n]^+ d_n}{\sum_n [i_n]^+}, \quad \bar{r} = \frac{\sum_n d_n}{\sum_n i_n}$
• The dual objective $\mathcal{L} = \mathcal{L}_{\text{tok}} + \lambda \mathcal{L}_{\text{deg}}$ is solved via dual ascent: $\lambda \leftarrow \max(0, \lambda + \alpha(\bar{r} - r^\star))$ If degradation exceeds tolerance $r^\star$, $\lambda$ automatically increases, forcing the router to retain more tokens. If the agent predicts actions easily, $\lambda$ decreases, aggressively pruning redundant tokens.

3. Joint Policy-Router Training

Because standard VLMs are pretrained on dense images, training the router alone fails (scoring only 4.9 on OSWorld). VTR trains the router and the language backbone $\pi_\theta$ jointly over 2,000 steps. The backbone learns to interpret sparse visual tokens, while the router learns task-specific visual saliency.

---

Key Quantitative Results

VTR was evaluated across three demanding benchmarks: OSWorld (desktop OS workflows), CUA-World (200 professional desktop applications), and the Markov-AI 500-hour gaming dataset (36 real-time video games at 30 FPS).

1. Benchmark Task Performance vs. Token Count

Method	Tokens per Image (OSWorld)	OSWorld Success Score $\uparrow$	Tokens per Image (CUA-World)	CUA-World Success Score $\uparrow$	Tokens per Frame (Gaming)	Gaming Action Accuracy $\uparrow$
Dense Full-Resolution	2,040	18.0	2,040	20.2	220	5.8%
Dense Half-Resolution (50% Area)	1,020	15.2	1,020	18.1	110	4.9%
Pixel-Diff Baseline	480	5.3	510	6.8	42	2.1%
H.264 Video Codec Baseline	620	9.4	680	11.2	58	3.4%
Frozen Router Baseline	510	4.9	530	5.8	38	2.6%
VTR (Ours, Block Anchor)	816 (2.5× fewer)	18.1 (+0.1)	703 (2.9× fewer)	20.1 (-0.1)	38 (5.8× fewer)	5.9% (+0.1%)
VTR Sliding (Ours)	790 (2.6× fewer)	17.9 (-0.1)	690 (3.0× fewer)	19.8 (-0.4)	35 (6.3× fewer)	5.8% (±0.0%)

2. Inference Efficiency & Cost Reductions

Metric	Dense Baseline	VTR (Desktop / OSWorld)	VTR (CUA-World)	VTR (Real-Time Video Games)
Visual Token Reduction	1.0× (None)	2.5× reduction	2.9× reduction	5.8× reduction
Prefill Latency Speedup	1.0× (Baseline)	2.3× faster	2.2× faster	2.9× faster
Total Inference Cost	1.0× (Baseline)	2.4× cheaper	1.8× cheaper	3.3× cheaper
Peak Memory Footprint	100%	-58% KV cache	-64% KV cache	-81% KV cache

Key Experimental Discoveries:

1. Zero Degradation at 2.5x–5.8x Token Pruning: VTR matches full-resolution dense models on OSWorld (18.1 vs 18.0) and gaming accuracy (5.9% vs 5.8%) while using up to 5.8× fewer visual tokens.
2. Superiority Over Downsampling & Heuristics: Lowering image resolution destroys small UI elements, dropping OSWorld score by 2.8 points. Rule-based pixel-diff and H.264 collapse to 5.3 and 9.4 points. VTR maintains full pixel fidelity on kept regions.
3. Adaptive Routing Across Motion Regimes: The router automatically adjusts token allocation without human tuning: keeping 26.3% of tokens on text-heavy desktop tasks, 5.9% in high-motion games, and only 2.7% in low-motion games.

---

Practical Implications

1. Deploying Real-Time Computer-Use Agents: By speeding up prefill by nearly 3× and curbing visual token explosion, VTR makes it feasible to run reactive desktop and gaming agents under tight real-time SLA constraints without falling behind the user interface.
2. Serving Optimization in Production: VTR runs natively inside high-throughput serving runtimes like vLLM. By dropping 60%–80% of visual tokens before language model forward passes, serving clusters can serve 2× to 3× higher concurrent user streams on identical GPU hardware.
3. Task-Relevant Delta Compression: Moving away from naive video codecs toward task-aware visual routing provides a blueprint for general multimodal agents operating over streaming video feeds.

                  计算机操作智能体 (CUA) 的视觉 Token 膨胀危机

  第 1 步              第 25 步             第 50 步              第 100 步
  [单帧截图: 2K] ──> [累积历史: 50K] ──> [累积历史: 100K] ──> [累积历史: 20万+ Tokens!]
         │                   │                    │                     │
  Prefill耗时: 120ms   Prefill耗时: 450ms   Prefill耗时: 980ms    Prefill耗时: 2,400ms (实时控制瘫痪!)
  单步成本: $0.01      单步成本: $0.15      单步成本: $0.45       单任务累积成本: 飙升至数美元

  核心现实：95% 的像素是静止不变的（任务栏、边距）或与决策完全无关的干扰（后台广告动画）。

                     视觉 TOKEN 动态路由 (VTR) 计算流水线

  多轮历史观测截图 [ o_{t-3}, o_{t-2}, o_{t-1}, o_t ]
       │
       ├─ o_{t-3}（锚点帧 Anchor）：100% 满分辨率完整保留全局基准
       │
       └─ 后续多轮截图 [ o_{t-2}, o_{t-1}, o_t ]：
              │
              ▼
       ┌─────────────────────────────────────────────────────────┐
       │   轻量时空路由网络 f_phi                                │
       │   跨时间追踪每个空间图块 j 的时序变化轨迹               │
       │   输出伯努利保留概率 p_{t, j}                           │
       └─────────────────────────────────────────────────────────┘
              │
              ├─ 静态任务栏 / 空白背景 ──> 剔除 (0 Tokens)
              ├─ 循环播放的背景视频广告 ──> 剔除 (0 Tokens)
              └─ 正在输入的文件名对话框 ──> 保留 (全分辨率 Token，附带原始空间位置编码)
              │
              ▼  (视觉 Token 数量急剧缩减 2.5x - 5.8x)
       ┌─────────────────────────────────────────────────────────┐
       │   多模态动作决策模型 pi_theta (vLLM 原生推理引擎)       │
       │   联合训练适配稀疏输入：精准预测下一步鼠标/键盘动作     │
       └─────────────────────────────────────────────────────────┘
              │
              ▼
       输出高精度键鼠操作（动作成功率与满分辨率密集模型完全打平！）