Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

Paper: 2606.12412 Authors: Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu Categories: cs.CV, cs.AI

The Gap

Multi-modal LLMs project images into hundreds of visual tokens, making decoder inference expensive.
Existing reduction methods (FastV, PDrop, Nüwa) all follow a rank-and-remove paradigm: compute an importance score (e.g. attention weight), keep the top-k tokens, and permanently discard the rest.

The paper exposes a hidden assumption: token importance is assumed to be static across decoder layers.
In reality, importance changes dynamically: a token that looks irrelevant in early layers may become critical for spatial grounding in later layers.
Permanent removal therefore irreversibly loses information that could have been useful later.

[ Problem: Rank-and-remove discards low tokens forever ]
        |
        | (Gap: token importance varies across depth)
        v
[ Assumption: low-scored tokens are always useless ]
        |
        | (This is fragile, especially for grounding tasks)
        v
[ Method: Replace removal with deferred routing ]
        |
        | (Deferred tokens re-enter candidate pool next stage)
        v
[ Evidence: Reroute improves grounding on LLaVA/Qwen   ]
[ under aggressive pruning, same compute budget       ]
        |
        v
[ Conclusion: token reduction should be recoverable routing ]

The Increment

One sentence: Before Reroute, token reduction meant irrevocable deletion; after Reroute, it becomes a multi-stage routing process where tokens can be deferred and re-evaluated.

Core Mechanism

Reroute is a training-free plug-in that sits on top of any existing token-reduction method that uses stage-wise decisions (e.g. every few transformer layers). At each routing stage, the method:

1. Scores all tokens in the current candidate pool using the same attention-score rule as the base method (e.g. average attention weight).
2. Selects a subset of high-score tokens to pass through the current decoder block (processing as usual).
3. Defers the remaining low-score tokens — they bypass the block entirely (no attention, no KV-cache update), but are preserved and merged back into the candidate pool at the next routing stage.
4. At the next stage, the pool now consists of: previously selected tokens (still active), previously deferred tokens (re-entering), and possibly new tokens if the base method re-introduces some.

This cycle repeats for each routing stage. The critical insight is that a token rejected early has a second chance later. Because the base method’s stage schedule and scoring rule are reused, the theoretical TFLOPs and KV-cache budget are unchanged — the same number of tokens pass through at each stage; only the identity changes.

[ ASCII internals: stage-by-stage routing ]

Stage n:
   Candidate Pool (C) -----> Attention Score Ranking
                              |
          +-------------------+-------------------+
          |                                       |
     High-score tokens                      Low-score tokens
     (pass through block)                   (bypass block)
          |                                       |
          v                                       v
     Stage n output (processed)          Deferred pool (stored)
          |                                       |
          +-------------------+-------------------+
                              |
                         n+1: merge with stage output
                              v
                    New Candidate Pool (C')
                    (previous deferred + previous output)

Imagine a restaurant with a multi-round seating policy.

• The head chef (scoring rule) rates each customer (token) based on current appetite (attention score).
• In each seating round (decoder stage), only high-rated customers are allowed to eat (pass through the block).
• Low-rated customers are sent to a waiting lounge (deferred pool) — they don’t eat yet, but they stay warm.
• When the next seating round begins, the chef reconsiders everyone: customers who just ate (still at their table) and waiting customers who are now hungrier.
• This gives previously rejected customers a chance to be seated later if their appetite grows.
• The restaurant never kicks anyone out permanently; it just delays service.
• The kitchen capacity (TFLOPs) stays the same because exactly the same number of diners are served per round; only the guests change.

Key Concepts

• Token importance dynamics: In standard VLMs, a visual token’s relevance to the current text query is not fixed across decoder layers. Early layers may focus on coarse layout; later layers need fine details for phrases like “the cup on the left”. A token that appears noisy early may become uniquely informative later.
  Concrete example: In an image with three cups, the “left cup” region produces tokens. In early decoder layers, attention is spread across all cups. By layer 20, only the left cup’s tokens are attended to. Under a rank-and-remove method that prunes at layer 5, the left-cup tokens might be discarded because they were mid-ranked early. Reroute keeps them alive.

• Deferred routing: The core operation is not “delete” but “delay”. Rather than making a one-shot decision, the method postpones the decision to a later stage, reusing the same scoring mechanism. This is possible because the decoder has multiple stages (layers), and you can choose to skip a token’s processing at one layer without erasing its representation.
  Mechanism: The deferred token is simply not passed through the attention and MLP of the current block. Its hidden state remains unchanged. When re-entering the pool, it gets a fresh chance at the next scoring.

• Budget-preserving reuse: The base method already has a fixed per-stage token budget (e.g., keep 50% of tokens every 4 layers). Reroute does not change that budget; it only changes *which tokens occupy those slots. Therefore, the compute and memory footprint (KV-cache, FLOPs) are identical to the base method. The only overhead is a tiny routing decision (score ranking per stage, which the base method already does). The gain comes from better token selection across time.

Framework Shift

Before (rank-and-remove):            After (Reroute):

[Stage 1]                            [Stage 1]
   Score all tokens                     Score all tokens
   Keep top-k -> pass                  Keep top-k -> pass
   Discard rest (delete)               Defer rest (store)
         |                                  |
[Stage 2]                            [Stage 2]
   New tokens only (maybe             Pool = stage1 passed +
   from image cropping etc.)          stage1 deferred + new
   Score, keep, discard               Score, keep top-k -> pass
         |                            Defer rest -> store
         |                                  |
[Later stages]                       [Later stages]
   No chance for discarded            Deferred tokens can be
   tokens to re-enter                 selected at any later stage

From irreversible removal to recoverable routing — the core shift is that tokens are not permanently deleted; they are given multiple opportunities across decoder depth.

Expert Assessment

Problem choice: Real and timely. Token reduction is a hot topic for efficient VLM inference, and the fragility of static importance has been occasionally noted but never directly addressed. The gap is genuine.

Method maturity: Clever and light-touch. Reroute is a small algorithmic tweak (change “remove” to “defer”) that logically plugs any existing stage-wise pruning method. It is not brute-force but an elegant reuse of existing infrastructure. One could argue that a simpler approach would be to use a more fine-grained scoring (e.g., per-head per-layer scores) — but that would change the budget. Reroute’s beauty is that it achieves improvement without altering compute assumptions.

Experimental integrity: The paper tests on two backbones (LLaVA-1.5, Qwen) and three base methods (FastV, PDrop, Nüwa). The key metric is grounding accuracy (e.g., referring expression segmentation) under aggressive reduction ratios (e.g., 80-90% tokens removed). The improvements are consistent but modest (a few points). A concern: they do not show how Reroute affects *in-distribution VQA performance under mild reduction — they should, because the base methods already degrade gracefully; Reroute might hurt slightly when reduction is mild. No red flags, but the paper would be stronger with more ablation on the choice of scoring rule and stage frequency.

Writing quality: Clear and well-structured. The abstract and introduction set up the problem precisely. The method section is well-illustrated. The weakness is the experimental section — the results are spread across many tables without a clear summary figure. A single “grounding gain vs. token budget” plot would instantly convey the benefit.

Verdict: weak accept — a clean, simple idea with solid evidence. It is worth attention if you work on efficient multi-modal LLMs or token compression.

Takeaways

For practitioners building real-time VLM applications:

1. Drop-in upgrade for existing pruning pipelines: If you already use FastV, PDrop, or any stage-wise token pruning, replace the “remove” step with “defer and re-enter”. Implementation is trivial (store a separate tensor for deferred tokens, concatenate at next stage). No retraining is needed.
2. Works best under aggressive compression: The advantage is largest when you are forced to keep very few tokens per stage (e.g., 10-20%). For mild compression (50%+), moving tokens between stages may not help much.
3. General principle for sequential compression: The idea of “deferral rather than deletion” may transfer to other sequential models (e.g., long-document encoding, video frame selection) where early decisions are uncertain. Any pipeline that makes discrete decisions per stage can benefit from a re-evaluation window.