VISOR: Teaching AI Agents to Search Visual Documents Without Getting Lost

Paper: 2604.09508 Authors: Yucheng Shen, Jiulong Wu, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao Categories: cs.CV, cs.AI

The Gap

Existing agentic Visual RAG systems (like ReAct-based approaches) can retrieve and reason over visual documents iteratively. They work fine for simple queries. But when you need multi-step reasoning across dozens of pages—like “compare the revenue trends in Q2 and Q4 slides, then find the corresponding product launch dates”—two things break down:

Visual Evidence Sparsity: The agent processes each page in isolation. Evidence on page 5 doesn’t connect to evidence on page 23. Fine-grained visual actions (zoom, crop, highlight) are available but misused—agents apply them randomly, degrading retrieval quality instead of improving it.

Search Drift: As the agent retrieves more pages, visual tokens pile up. The context window fills with raw image embeddings. By step 15, the agent has forgotten what it was originally looking for. It starts chasing tangents, retrieving irrelevant pages, and the reasoning chain collapses.

Prior work (AutoRAG, VisRAG) assumes short horizons or pre-filtered documents. They don’t handle the cognitive overload of long visual search trajectories.

Problem: Multi-step visual reasoning across 50+ pages
   |
   v
Assumption: Agents need (1) cross-page evidence linking
            (2) protection from context dilution
   |
   v
Method: VISOR = Evidence Space + Action Correction + Sliding Window
   |
   v
Evidence: ViDoSeek (+8.2%), SlideVQA (+6.7%), MMLongBench (+5.3%)
   |
   v
Conclusion: Structured evidence + dynamic context > raw token accumulation

The Increment

One sentence: Before VISOR, agentic visual RAG drowned in its own retrieved context after 10-15 steps; after VISOR, agents maintain coherent search objectives across 30+ retrieval steps by anchoring evidence and discarding raw visual tokens.

Core Mechanism

VISOR has three interlocking components. First, an Evidence Space—a structured memory that stores extracted claims from each retrieved page, not raw images. When the agent retrieves page 12, it doesn’t just append the image to context; it extracts “Q2 revenue: $45M” and links it to earlier evidence. This enables cross-page reasoning without re-processing images.

Second, a Visual Action Evaluator—before executing a visual action (zoom, crop), the agent predicts whether it will improve retrieval. If the prediction is low-confidence, the action is corrected or skipped. This prevents the “random zoom” problem where agents waste steps on useless visual manipulations.

Third, a Dynamic Trajectory with Sliding Window—the agent’s context has two parts: (1) the Evidence Space (persistent, grows slowly), (2) a sliding window of the last N raw interactions (discarded as new ones arrive). This keeps the context window from exploding with visual tokens. Additionally, at each step, the system re-injects the original query intent to prevent drift.

Query --> Agent --> Action Evaluator --> Retrieve Page
            |              |                    |
            v              v                    v
      [Evidence Space] <-- Extract Claims <-- Raw Image
            |                                   |
            v                                   v
      Cross-page      Sliding Window --> Discard Old Tokens
      Reasoning            |
                           v
                    Next Action (with Intent Re-injection)

Think of VISOR like a detective investigating a cold case across multiple file cabinets. The Evidence Space is the detective’s corkboard—photos, notes, red strings connecting clues across different files. The detective doesn’t carry every file folder around (that’s the raw visual tokens). Instead, they extract key facts and pin them to the board.

The Action Evaluator is the detective’s partner who says “wait, before you zoom in on that blurry receipt, are you sure it’s relevant?” preventing wasted effort. The Sliding Window is the detective’s short-term memory—they remember the last few files they opened, but discard the details of files from an hour ago to avoid confusion. The Intent Re-injection is the detective periodically checking the original case file: “Right, I’m looking for the suspect’s alibi on March 15th, not their tax records.”

Without the corkboard, the detective would re-read every file at each step. Without the partner, they’d waste time on dead ends. Without clearing short-term memory, they’d drown in details. Without checking the case file, they’d forget what crime they’re solving.

Key Concepts

• Evidence Space: Instead of keeping raw retrieved images in context (which consumes thousands of tokens per page), VISOR extracts structured claims—short text snippets like “Slide 5: Product X launched in Q2” or “Page 12: Revenue chart shows 15% decline.” These claims are stored in a persistent memory that grows slowly (10-20 tokens per page instead of 500-1000). When the agent needs to reason across pages, it queries the Evidence Space, not the raw images. This is like taking notes during a lecture instead of recording the entire video—you lose some detail but gain the ability to review 50 lectures without drowning in footage.

• Search Drift: In long reasoning chains, agents accumulate context. By step 20, the context window contains 15 retrieved images, intermediate reasoning steps, and action logs. The original query (“find the correlation between marketing spend and user growth”) is buried under 30,000 tokens of visual embeddings. The agent starts retrieving pages about unrelated topics because the signal-to-noise ratio has collapsed. VISOR prevents this by (1) discarding old raw interactions via sliding window, (2) re-injecting the original query at each step to re-anchor the agent’s objective. It’s like a GPS that periodically recalculates your route instead of just appending every turn you’ve made.

• Visual Action Evaluation: Agents have access to visual tools—zoom into a chart, crop a table, highlight a region. But when should you use them? Existing systems let agents call these actions freely, leading to “action spam”—zooming into irrelevant areas, cropping empty space. VISOR adds a prediction step: before executing a visual action, the agent estimates “will this improve my retrieval quality?” If confidence is low, the action is corrected (e.g., “zoom into the chart title instead of the legend”) or skipped. This is trained via reinforcement learning with credit assignment—actions that led to successful retrieval get positive reward, useless actions get negative reward.

Framework Shift

Before (ReAct-style VRAG):              After (VISOR):

Query                                   Query
  |                                       |
  v                                       v
Retrieve Page 1 --> [Image Tokens]     Retrieve Page 1 --> Extract Claims
  |                        |              |                      |
  v                        v              v                      v
Reason --> Retrieve Page 2 --> [More Tokens]   Reason <--> [Evidence Space]
  |                        |              |                      |
  v                        v              v                      v
Reason --> Retrieve Page 3 --> [Even More...]   Retrieve Page 2 --> Add Claims
  |                        |              |                      |
  v                        v              v                      v
[Context Overload]    [Drift]          Sliding Window      Intent Re-inject
  |                        |              |                      |
  v                        v              v                      v
Wrong Answer          Lost Objective   Retrieve Page 3 --> Correct Answer

Raw accumulation                       Structured memory + dynamic pruning

[One sentence: From accumulating raw visual tokens until context collapse, to maintaining a structured evidence layer while dynamically pruning interaction history.]

Expert Assessment

Problem choice: Real gap. Multi-step visual reasoning over long documents is a legitimate bottleneck in enterprise RAG systems (financial reports, legal documents, technical manuals). The paper correctly identifies that existing agentic RAG assumes short horizons or text-only retrieval. Not manufactured.

Method maturity: Mix of clever and brute force. The Evidence Space and sliding window are elegant—they address the core problem (context explosion) with minimal overhead. The Visual Action Evaluator feels like a patch—it’s solving a problem created by giving agents too many tools in the first place. A simpler approach might be to restrict visual actions to high-confidence scenarios from the start, rather than predict-then-correct. The GRPO-based RL training is heavy machinery; unclear if supervised fine-tuning on curated trajectories would achieve 80% of the gains with 20% of the complexity.

Experimental integrity: Baselines are fair (ReAct, AutoRAG, VisRAG). Numbers are solid—consistent gains across three benchmarks (ViDoSeek, SlideVQA, MMLongBench). The ablation studies are thorough, showing each component contributes. One red flag: the paper doesn’t report failure modes or error analysis. What happens when the Evidence Space extracts wrong claims? How often does Intent Re-injection fail to prevent drift? The writing glosses over these.

Writing quality: The method section is dense—too many acronyms (GRPO, VRAG, DT-SW-II) without enough intuition-building. The paper would benefit from a running example: “Here’s a query, here’s how VISOR processes it step-by-step, here’s where baseline methods fail.” The related work section is thorough but reads like a literature dump. The experimental section is strong.

Verdict: weak accept — Solid contribution to a real problem, but the method feels over-engineered and the writing doesn’t make the core insights accessible enough. The Evidence Space idea is worth stealing; the rest is incremental.

Takeaways

Structured memory over raw accumulation: If you’re building any multi-step retrieval system (visual or text), don’t just append retrieved content to context. Extract structured claims, store them separately, and query them. This applies beyond visual RAG—think code search, database query optimization, even conversational agents that need to remember facts across long dialogues.

Dynamic context pruning: The sliding window + intent re-injection pattern is generalizable. In any long-horizon agent system, periodically discard low-value context and re-inject the original objective. This prevents drift without requiring the agent to “remember everything.”

Action evaluation as a design pattern: Before letting an agent execute a tool call, add a prediction step: “will this action improve my outcome?” Train this via RL or supervised learning. This reduces action spam in agentic systems—applicable to web agents, code agents, robotics.

Credit assignment for RL in dynamic contexts: The paper’s GRPO training with state masking is a technical trick worth noting. When your agent’s context changes dynamically (things get added/removed), standard RL credit assignment breaks. VISOR masks out the dynamic parts during reward computation, focusing credit on the persistent state (Evidence Space). This is useful for any RL system with non-stationary state representations.