ESI-Bench: Teaching AI to Explore Before It Reasons

Paper: 2605.18746 Authors: Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin Choi Categories: cs.CV, cs.AI, cs.CL, cs.LG, cs.RO

The Gap

Existing spatial reasoning benchmarks assume oracle observations—agents receive perfect views and reason about what they see. This mirrors how we evaluate vision models: feed them images, measure accuracy. But real spatial intelligence isn’t passive perception. A child learning object permanence doesn’t stare at a static scene; they peek behind the couch, shake the box, open the drawer. The observation itself is the outcome of action.

Prior work (Embodied Question Answering, Vision-and-Language Navigation) gives agents mobility but still treats perception as given. The agent moves, but the task design doesn’t require reasoning about what to look at versus what the view reveals. The gap: no benchmark tests whether models can close the perception-action loop—decide what abilities to deploy (look, walk, grab), sequence them strategically, and accumulate evidence through exploration rather than hoping the right view appears.

Problem: Spatial tasks need active evidence gathering
   |
   v
Assumption: Oracle views hide action-selection competence
   |
   v
Method: ESI-Bench forces agents to choose actions that yield informative observations
   |
   v
Evidence: Active exploration >> passive multi-view; failures trace to poor action choice
   |
   v
Conclusion: Action blindness, not perception weakness, bottlenecks spatial intelligence

The Increment

One sentence: Before—spatial reasoning assumed perfect observations; after—spatial reasoning requires learning what to observe.

Core Mechanism

ESI-Bench is built on OmniGibson, a physics simulator supporting perception, locomotion, and manipulation. It spans 10 task categories grounded in Spelke’s core knowledge systems: object permanence (tracking occluded objects), solidity (understanding physical constraints), containment (reasoning about what’s inside containers), support (predicting stability), and shape constancy (recognizing objects across viewpoints). Each task withholds critical information from the initial view—an object is behind a wall, inside a closed drawer, or only identifiable from a specific angle.

The agent receives no privileged state. It must decide: Should I move closer? Open this container? Rotate the object? Each action costs tokens (simulating real-world effort) and yields a new observation. The benchmark measures not just final accuracy but the efficiency of the exploration strategy—how many actions to reach a confident, correct answer.

Experiments compare three regimes: (1) single passive view, (2) random multi-view (agent sees N random perspectives), (3) active exploration (agent chooses actions). State-of-the-art MLLMs (GPT-4V, Gemini, Claude) are tested with and without explicit 3D grounding (point clouds, depth maps).

Task Setup:
  Initial View --> [Occluded / Ambiguous Scene]
       |
       v
  Agent Decision Tree:
    - Perceive (zoom, rotate view)
    - Locomote (move to new position)
    - Manipulate (open, lift, shake)
       |
       v
  New Observation --> Update Belief --> Repeat or Answer
       |
       v
  Evaluation: Accuracy + Action Efficiency

Think of ESI-Bench as a detective game where the crime scene is interactive. A passive detective gets a single photo and guesses. A random detective gets 20 photos from arbitrary angles—more data, but no strategy. An active detective walks the scene, opens drawers where evidence might hide, checks sightlines, and tests hypotheses. The crime (task) is the same, but the investigation method (observation strategy) determines success. ESI-Bench is the crime scene; the agent is the detective; actions are investigative moves. The benchmark doesn’t test “Can you see the clue?” but “Do you know where to look for it?”

Key Concepts

• Perception-Action Loop: In passive vision, perception is input and reasoning is output. In embodied intelligence, perception and action form a cycle: you act to change what you perceive, then perceive to decide the next action. A robot determining if a box is empty can’t solve this from a photo—it must lift the box (action) to feel its weight (perception), or open it (action) to see inside (perception). The loop closes when action choices are informed by what you need to perceive, not just what you currently see. ESI-Bench operationalizes this by making task success depend on choosing actions that reveal hidden information, not just processing visible information better.

• Action Blindness: Models fail not because they can’t recognize objects or estimate depth, but because they can’t map uncertainty to action. When a model sees a closed drawer and needs to know what’s inside, action blindness manifests as: (a) not recognizing the drawer is closed, (b) recognizing it but not choosing to open it, or (c) opening it but not understanding that the new view resolves the uncertainty. It’s a metacognitive failure—the model doesn’t know what it doesn’t know, so it can’t seek the right evidence. Human studies in the paper show people naturally seek falsifying viewpoints (“Let me check the other side to be sure”). Models commit early with high confidence regardless of evidence quality.

• Emergent Spatial Strategies: Without explicit instructions, agents discover task-specific exploration patterns. For object permanence tasks, successful agents learn to circle occluders. For containment tasks, they learn to manipulate containers before answering. These strategies aren’t programmed—they emerge from the reward structure (correct answers after fewer actions score higher). The emergence is fragile: random multi-view often degrades performance because irrelevant views add noise, proving that *more observation ≠ better reasoning without strategic selection.

Framework Shift

Before (mainstream approach):        After (this paper):

  [Image] --> [Model] --> [Answer]     [Scene] --> [Agent]
                                              |
                                              v
                                        Choose Action
                                              |
                                              v
                                        [New View] --> Update
                                              |
                                              v
                                        Repeat or Answer

  Perception = Input                   Perception = Outcome of Action
  Task: Reason about what you see      Task: Decide what to see

From oracle observations to earned observations, the core shift is reframing the observer as an actor.

Expert Assessment

Problem choice: This is a real gap. Embodied AI research has focused on navigation and manipulation as separate modules, but the integration—using action to resolve perceptual uncertainty—remains underexplored. The problem sits at the intersection of vision, robotics, and reasoning, which makes it both important and difficult to benchmark cleanly. The grounding in Spelke’s core knowledge systems is intellectually honest; these aren’t arbitrary tasks but foundational spatial competencies.

Method maturity: The benchmark design is sound. Using OmniGibson provides physical realism without the cost of real-world data collection. The action space (perceive/locomote/manipulate) is minimal but sufficient to test the hypothesis. The comparison between passive, random multi-view, and active exploration is the right ablation. However, the paper leans heavily on prompting existing MLLMs rather than training embodied agents from scratch, which limits the conclusions about what’s learnable versus what’s a prompting artifact.

Experimental integrity: Baselines are fair. The finding that random multi-view often hurts performance is surprising and well-documented. The human studies add credibility—showing humans naturally seek falsifying evidence while models don’t is a clean result. One concern: the 3D grounding experiments show imperfect 3D representations harm performance, but the paper doesn’t deeply investigate *why. Is it noise in the point clouds? Misalignment between 2D and 3D? This feels like a missed opportunity for a follow-up ablation.

Writing quality: The abstract and introduction are strong. The related work section is thorough but dense—cutting 20% would improve flow. The results section buries the lead: the action blindness finding (Section 4.3) should come earlier because it’s the paper’s most important contribution. The human study (Section 4.4) is well-executed but feels tacked on; integrating it into the main narrative would strengthen the argument.

Verdict: Strong accept—this benchmark will be widely used, and the action blindness insight reframes how we should evaluate spatial reasoning.

Takeaways

For practitioners building embodied agents: Don’t assume more views = better reasoning. Test whether your agent can *choose informative actions, not just process more data. If your agent explores randomly, you’re likely adding noise.

For vision researchers: Spatial reasoning benchmarks that provide oracle views are measuring the wrong thing. If your model scores 90% on a static spatial reasoning dataset, that doesn’t predict embodied performance—action selection is a separate competence.

For prompt engineers: The paper shows that prompting MLLMs to “explore actively” yields emergent strategies without explicit instructions. This suggests that task framing (emphasizing uncertainty and evidence-gathering) can unlock latent capabilities in foundation models.

Transferable technique: The “action cost” mechanism (each action consumes tokens, forcing efficiency) is a simple way to prevent brute-force exploration in any interactive benchmark. Steal this for your own evaluation setups.