WebEye: When Visual Perception Needs to Google First

Paper: 2605.12497 Authors: Bokang Yang, Xinyi Sun, Kaituo Feng, Xingping Dong, Dongming Wu, Xiangyu Yue Categories: cs.CV

The Gap

Existing visual grounding and segmentation systems assume the target object’s identity is either directly visible in the image or already encoded in the model’s frozen knowledge. You point at a photo and say “find the red car” — the system knows what “red” and “car” mean, and the car is right there. This works for common objects and straightforward descriptions.

But what if you ask “find the company logo whose CEO just resigned yesterday”? The decisive evidence isn’t in the pixels. The model needs to first search the web for recent news, resolve which company that is, learn what their logo looks like, then return to the image and locate it. Prior work in referring expression comprehension, open-vocabulary detection, and visual question answering all operate within the closed-world assumption: everything needed to identify the target is either in the image or in pre-trained weights. They don’t handle knowledge-intensive queries requiring external fact retrieval, temporal reasoning about recent events, or multi-hop inference chains.

Problem: "Find X in this image"
   |
   v
Existing assumption:           This paper's reality:
X is visible + known    vs.    X requires external resolution
   |                              |
   v                              v
Direct perception              Search -> Resolve -> Perceive
   |                              |
   v                              v
Works for common objects       Handles knowledge-intensive queries

The Increment

One sentence: Before this paper, visual perception stopped at the image boundary; after it, perception can reach through the web to resolve hidden identities before grounding them in pixels.

Core Mechanism

WebEye introduces two components: a benchmark and a method. The benchmark, called WebEyes (plural), contains 120 images with 473 annotated objects where identification requires external knowledge — things like “the flag of the country that won the most recent World Cup” or “the logo of the company mentioned in today’s tech news.” Each query comes with verifiable evidence chains, precise bounding boxes or segmentation masks, and three task formulations: Search-based Grounding (output a box), Search-based Segmentation (output a mask), and Search-based VQA (output a grounded answer).

The method, Pixel-Searcher, is an agentic workflow with three stages. First, evidence acquisition: given a query and image, it decomposes the query into search-friendly sub-questions, retrieves web documents, and extracts relevant facts. Second, identity resolution: it synthesizes the retrieved evidence to determine what the target actually is (e.g., “Argentina won the 2022 World Cup, so the target is the Argentine flag”). Third, visual instance binding: it takes the resolved identity back to the image and uses a vision-language model to localize the corresponding object via box, mask, or answer generation.

Input: Image + Query ("Find the logo of X's parent company")
   |
   +---> [Evidence Acquisition]
   |        |
   |        +-> Decompose query into sub-questions
   |        +-> Web search for each sub-question
   |        +-> Extract relevant facts from documents
   |        |
   |        v
   |     Retrieved facts: "X was acquired by Y in 2023"
   |
   +---> [Identity Resolution]
   |        |
   |        +-> Synthesize facts into target identity
   |        |
   |        v
   |     Resolved identity: "Y's logo"
   |
   +---> [Visual Instance Binding]
            |
            +-> Locate "Y's logo" in image
            +-> Generate box / mask / answer
            |
            v
         Output: Grounded result

Think of Pixel-Searcher as a detective solving a case. The query is a cryptic clue (“find the person who owns the car parked outside the building where the suspect works”). A regular vision system is like a detective who only looks at the crime scene photo — if the answer isn’t visible, they’re stuck. Pixel-Searcher is a detective who can leave the photo, go to the library (web search), read through files (evidence acquisition), piece together the story (identity resolution), then return to the photo with the answer in hand (visual binding). The evidence acquisition stage is the library research — gathering scattered facts. Identity resolution is the “aha” moment when the facts click into a coherent answer. Visual binding is returning to the crime scene photo and pointing at the right person. The key insight: you can’t skip the library trip. The photo alone doesn’t contain enough information.

Key Concepts

• Perception Deep Research: Traditional visual perception assumes the target’s identity is self-evident from the image or model knowledge. Perception Deep Research formalizes the setting where identifying the target requires external fact retrieval and reasoning before perception can proceed. It’s the difference between “find the red car” (identity is given) and “find the car owned by the person who won yesterday’s lottery” (identity must be resolved externally first). The “deep research” part means the system must autonomously navigate knowledge sources, synthesize multi-hop inferences, and handle temporal or long-tail entities that weren’t in the training data.

• Agentic Search-to-Pixel Workflow: Instead of end-to-end neural models that try to learn everything, this approach treats the problem as a multi-stage pipeline where each stage uses specialized tools. “Agentic” means the system makes decisions about what to search, how to decompose queries, and when it has enough evidence — it’s not just a fixed sequence of operations. The workflow explicitly separates knowledge acquisition (search) from visual grounding (pixels), allowing each to use its best tools: search engines and LLMs for knowledge, vision models for localization. This modularity also makes failures interpretable: you can see whether the system failed because it couldn’t find the right web page, misunderstood the evidence, or couldn’t locate the object in the image.

Framework Shift

Before (closed-world perception):        After (open-world perception):

Query: "Find X"                          Query: "Find X"
    |                                        |
    v                                        v
Image + Model knowledge                  Image + Model + Web
    |                                        |
    |                                        +---> Search engine
    |                                        |
    |                                        +---> Evidence synthesis
    |                                        |
    v                                        v
Direct grounding                         Resolved identity -> Grounding
    |                                        |
    v                                        v
Box / Mask / Answer                      Box / Mask / Answer

Assumption: X is visible                 Assumption: X may be hidden
Failure mode: Unknown objects            Failure mode: Bad evidence or binding

From closed-world to open-world, the core shift is making perception conditional on external knowledge resolution rather than assuming all necessary information is pre-encoded.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The authors correctly identify that most vision benchmarks test recognition of things the model has seen before or can infer from visual context alone. Real-world applications — monitoring news events in images, identifying entities from recent developments, grounding references that require domain knowledge — routinely hit this limitation. The problem sits at the intersection of vision and language, where recent work has focused on scaling up pre-training or adding more modalities, but hasn’t seriously tackled the open-world knowledge bottleneck. The timing is right: search-augmented generation is maturing in NLP, and this paper asks the natural next question for vision.

Method maturity: Pixel-Searcher is more of a well-engineered pipeline than a novel algorithmic contribution. The three-stage decomposition (search, resolve, bind) is sensible but not surprising — it’s the obvious way to structure the problem once you’ve decided to use external search. The agentic aspect (query decomposition, iterative search) relies on existing LLM capabilities rather than introducing new mechanisms. That said, the engineering is non-trivial: getting search, evidence extraction, and visual grounding to cooperate reliably is harder than it looks. The method’s value is in demonstrating that this pipeline works and in establishing a baseline for future work. A simpler approach might skip the explicit identity resolution stage and directly feed retrieved documents to a vision-language model, but the authors’ ablations suggest the structured pipeline helps.

Experimental integrity: The benchmark design is careful. The authors manually curated queries requiring external knowledge and verified that evidence chains are traceable. The 120 images and 473 objects are small by modern standards, but the focus is on hard cases rather than scale. The three task views (grounding, segmentation, VQA) provide good coverage of how this problem manifests in practice. Baselines include recent vision-language models (GPT-4V, Gemini) and open-source alternatives, which is fair. The failure analysis is honest: they break down errors into evidence acquisition (search failed), identity resolution (wrong inference), and visual binding (couldn’t locate the object), showing that all three stages contribute to failures. One weakness: the paper doesn’t compare against a simpler baseline that just concatenates all retrieved documents and feeds them to a VLM without the structured resolution stage. This would help isolate the value of the agentic workflow.

Writing quality: The abstract and introduction are clear, but the method section is dense and could benefit from more examples walking through the full pipeline on a single query. The benchmark description is thorough, but the statistics (120 images, 473 objects, 645 QA pairs, 1,927 task samples) are presented without enough context — it’s hard to assess whether this scale is sufficient without seeing the diversity of query types. The related work section does a good job positioning the paper but could be more concise. The failure analysis in the experiments is the strongest part of the writing — it’s specific and actionable. If I were revising, I’d expand the method section with a detailed walkthrough and compress the related work.

Verdict: weak accept — Solid problem formulation and a useful benchmark, but the method is more engineering than insight, and the experimental scale is limited.

Takeaways

The key transferable idea is staged resolution with explicit intermediate representations. Instead of asking a model to do everything end-to-end (understand the query, search, reason, ground), break the task into stages where each stage produces a concrete intermediate output (retrieved facts, resolved identity, grounded box). This makes the system debuggable and lets you swap in better components as they become available. For practitioners building systems that need to combine retrieval and perception, the lesson is: don’t try to train a single model to do both — use a pipeline where retrieval and reasoning happen in language space, then hand off a resolved target to the vision model.

Another takeaway: benchmark design for knowledge-intensive tasks. The authors’ approach — manually curating hard cases, annotating evidence chains, providing multiple task views — is a template for building benchmarks in other domains where external knowledge is required. If you’re working on a problem where the model needs to “look things up,” this paper shows how to structure a benchmark that tests that capability rather than just memorization.