Seek to Segment: Teaching Agents to Actively Search 360° Scenes for Objects

Paper: 2607.02497 Authors: Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang Categories: cs.CV

The Gap

Existing referring segmentation models (LISA, GLaMM, etc.) treat the problem as a static image task: you hand them a photo and a text description, they hand back a mask. This works fine when someone else has already framed the shot for you. But embodied agents don’t live in that world. An agent standing in a room only sees a narrow field of view — to find “the red cup on the left shelf,” it must actively rotate, scan the environment, decide what it has already seen, and figure out where to look next. No prior work bridges this gap between referring segmentation and active visual exploration in panoramic settings.

The paper identifies a clean logical chain: fixed-perspective assumption limits real-world deployment → need an agent that explores 360° environments with memory → build a system (PanoSeeker) that incrementally constructs spatial memory and plans efficient search → validate on a new benchmark with expert trajectories and RL post-training → show substantial gains in both search efficiency and mask quality.

Problem
  |   Referring segmentation assumes fixed camera, passive input
  v
Gap
  |   Embodied agents see partial views, must ACTIVELY explore 360 deg
  v
Assumption
  |   Incremental spatial memory + VLM reasoning can enable efficient search
  v
Method
  |   PanoSeeker: VLM + EgoSphere memory + active viewpoint planning
  v
Evidence
  |   New APRS benchmark, expert-annotated trajectories, SFT + RL training
  v
Conclusion
  PanoSeeker significantly outperforms adapted baselines in efficiency and accuracy

The Increment

One sentence: Before this paper, referring segmentation was a “look once and segment” game; after this paper, it becomes an active exploration problem where agents must reason about what they’ve seen, where to look next, and when to stop — and there’s a benchmark, dataset, and trained model to prove it.

Core Mechanism

PanoSeeker operates in three phases: observe, remember, and act. At each timestep, the agent captures a local view from its current viewing direction. This view — along with the user’s language instruction — is fed into a Vision-Language Model (VLM) that reasons about whether the target object is visible and, if not, which direction to look next.

The key architectural contribution is EgoSphere, an explicit spatial memory structure. Think of it as a partial 360° panorama that the agent builds up over time. Each time the agent looks at a new direction, the local observation is projected onto the sphere at the corresponding azimuth and elevation coordinates. This means the agent doesn’t just remember “I saw a chair at timestep 3” — it maintains a unified spatial representation where every piece of visual information is pinned to a location in 360° space. As the agent explores, EgoSphere fills in, giving the VLM progressively richer context for planning.

Once the target is located, the agent performs active viewpoint alignment — it adjusts its viewing direction to center the target and then generates a segmentation mask. The entire pipeline is trained in two stages: supervised fine-tuning on expert-annotated search trajectories (with memory timelines showing what the agent should have seen and decided at each step), followed by reinforcement learning post-training that explicitly optimizes for search efficiency — penalizing redundant views and rewarding faster target localization.

User Instruction ("find the red cup")
            |
            v
  +-------------------+
  |   VLM (Reasoning) |<--------+
  +-------------------+         |
            |                   |
            v                   |
  +-------------------+         |
  | Viewpoint Planner |         |
  | (select delta-theta,        |
  |  delta-phi)         |         |
  +-------------------+         |
            |                   |
            v                   |
  +-------------------+         |
  |  Local View Capture         |
  +-------------------+         |
            |                   |
            v                   |
  +-------------------+         |
  |     EgoSphere     |         |
  |  (spatial memory  |---------+
  |   on 360 deg)     |
  +-------------------+
            |
     (target found?)
            |
            v
  +-------------------+
  |  Active Alignment |
  |  + Segmentation   |
  +-------------------+
            |
            v
       Mask Output

Here’s the metaphor that makes this click: imagine you’re a museum visitor looking for a specific painting, but you’re blindfolded except for a small peephole. At each moment, you can rotate your body and peek through the hole at one narrow slice of the gallery. You can’t see the whole room at once. But you’re not helpless — you carry a mental canvas (EgoSphere) where you paint in each slice as you see it, building up a growing panorama in your mind. Your reasoning ability (the VLM) looks at this partial panorama and decides: “I’ve covered the left wall, nothing there — let me try the right side.” When your canvas finally shows the painting, you center your peephole on it and trace its outline (segmentation). Without the mental canvas, you’d just spin randomly. With it, you plan efficiently, never re-examining the same wall twice. The RL training is like a coach watching you search and saying “you looked at that corner already, stop wasting time.”

Key Concepts

• Active Panoramic Referring Segmentation (APRS): Traditional referring segmentation is passive — someone hands you a complete image and says “segment the red cup.” APRS is active — you’re an agent standing in a 360° environment with only a narrow field of view, and you must physically rotate to find and segment the cup yourself. The key difference: you don’t start with all the information. You must earn it through exploration. It’s the difference between doing a jigsaw puzzle with the box lid in front of you versus doing it blindfolded, only seeing one piece at a time.

• EgoSphere: An explicit spatial memory that maps what you’ve seen onto a spherical coordinate system. If you’re an agent at the center of a sphere, every time you look in a direction, the visual information gets “painted” onto the corresponding spot on the sphere’s inner surface. Over time, you build a partial panorama. The “Ego” part means it’s centered on the agent — it’s YOUR view of the world, stitched together from YOUR sequential observations. This is different from a generic replay buffer because it preserves spatial relationships: you know that what you saw at azimuth 45° is to the right of what you saw at azimuth 0°.

• Search Trajectory Dataset with Memory Timelines: To train the agent, they didn’t just collect final answers. They recorded expert demonstrations that include, at every timestep, what the agent saw, what was stored in memory, and what decision was made (keep looking left? turn right? stop and segment?). This “memory timeline” annotation lets them do supervised fine-tuning on the entire decision process, not just the end result. It’s like teaching someone to cook not by showing them only the finished dish, but by recording a master chef’s every move and thought.

Framework Shift

Before (mainstream approach):          After (this paper):

  Full Image                            360 deg Environment
       |                                      |
       v                                      v
  +----------+                        +----------------+
  | RefSeg   |                        | Active Agent   |
  | Model    |                        | (partial view) |
  +----------+                        +----------------+
       |                                      |
       v                               observe / remember / plan
    Mask Output                         (iterative loop)
                                               |
                                               v
                                      +----------------+
                                      | EgoSphere      |
                                      | (spatial memory)|
                                      +----------------+
                                               |
                                               v
                                      Mask Output
                                         (after search)

Assumption: all info given              Assumption: must explore
Action: none (passive)                  Action: rotate, scan, decide
Memory: none needed                     Memory: essential (EgoSphere)

From passive single-shot segmentation to active memory-augmented search, the core shift is treating object grounding as a sequential decision problem in spatially continuous environments rather than a one-shot pattern matching task.

Expert Assessment

Problem choice: This is a genuine and well-identified gap. Embodied AI is moving fast, and the disconnect between static perception benchmarks and agents that actually need to act in environments is a real pain point. APRS is a natural next step that cleanly combines referring segmentation with active perception. It’s not manufactured — if you think about deploying a robot assistant, this task is exactly what it needs to do. Good positioning in the field’s trajectory.

Method maturity: The EgoSphere idea is the standout contribution — it’s a clean, interpretable spatial memory that gives the VLM useful structure rather than just feeding it raw history. The SFT-then-RL training pipeline is solid but not novel in its components (this is becoming standard for agent training). One concern: the method is heavily dependent on the VLM’s spatial reasoning ability, and there’s a risk that as VLMs get better at handling long contexts, explicit spatial memory might become less necessary. The approach is clever engineering rather than a fundamental algorithmic breakthrough — but sometimes that’s exactly what’s needed.

Experimental integrity: The creation of a new benchmark with expert-annotated trajectories is valuable but also a red flag — when you build both the test and the model, there’s always risk of distributional bias favoring your approach. They do adapt several strong baselines (including VLMs with different prompting strategies), which is good. However, I’d want to see: (1) how much of the gain comes from EgoSphere vs. just the RL post-training, (2) sensitivity to VLM backbone choice, and (3) failure case analysis. The numbers are convincing but not bulletproof.

Writing quality: The paper reads clearly and the task definition is crisp. The supplementary materials likely carry a lot of the weight here. Where they cut corners: the RL reward design feels underexplored — it’s mentioned as optimizing “search efficiency” but the exact formulation and ablation of reward components deserves more space. A deeper analysis of failure modes (when does the agent get stuck in loops? when does EgoSphere accumulate errors?) would significantly strengthen Section 4.

Verdict: weak accept — The task definition is timely and the EgoSphere mechanism is a genuinely useful idea, but the method is incremental in its components and the evaluation could be more rigorous on ablations and failure analysis.

Takeaways

Three concrete ideas you can steal:

1. Spatial memory as a spherical projection for active agents: If you’re building any agent that explores environments with a limited field of view, maintaining an explicit ego-centric spatial map (not just a flat buffer of past frames) dramatically improves planning. The EgoSphere idea transfers directly to robot navigation, AR/VR exploration, or even drone inspection tasks.

2. Memory timeline annotations for agent training: Instead of just labeling “correct final action,” annotating the entire decision trajectory — what the agent should know and decide at each step — enables much more effective supervised pre-training before RL. If you’re training any sequential decision agent, consider this two-level annotation strategy.

3. RL post-training on top of SFT for efficiency optimization: This is becoming a pattern (it echoes the RLHF pipeline): supervised fine-tuning teaches the agent what good behavior looks like, then RL explicitly optimizes the metric you actually care about (here, search efficiency). If your downstream task has a measurable efficiency or cost metric, this two-stage recipe is worth trying before jumping to end-to-end RL from scratch.