CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation

Paper: 2602.18424 Authors: Xia Su, Ruiqi Chen, Benlin Liu, Jingwei Ma, Zonglin Di, Ranjay Krishna, Jon Froehlich Categories: cs.CV, cs.RO

Abstract

Vision-Language Models have demonstrated impressive capabilities in Vision-Language Navigation tasks, but their ability to reason about agent-specific mobility constraints remains largely unexplored. This paper introduces CapNav, a novel benchmark that evaluates VLMs on capability-conditioned indoor navigation. The benchmark encompasses 45 real-world indoor scenes, 473 navigation tasks, and 2365 question-answer pairs across five distinct agent types—from wheelchair users to quadruped robots—each with unique physical dimensions and mobility capabilities. Evaluation of 13 state-of-the-art VLMs reveals a critical limitation: navigation performance degrades significantly as mobility constraints become more restrictive, with models particularly struggling on obstacles requiring spatial dimensional reasoning.

Key Contributions

• Introduction of Capability-Conditioned Navigation (CapNav), the first benchmark specifically designed to evaluate VLMs on navigation tasks conditioned by agent-specific physical and operational capabilities
• A comprehensive dataset featuring 45 real-world indoor environments, 473 carefully designed navigation tasks, and 2365 QA pairs spanning five representative agent types with distinct mobility profiles
• Systematic evaluation of 13 modern VLMs, revealing fundamental limitations in capability-aware spatial reasoning and obstacle assessment
• Empirical evidence showing that navigation performance drops sharply as mobility constraints tighten, with even advanced models failing on basic dimensional reasoning tasks

Benchmark Design and Methodology

CapNav addresses a critical gap in VLM evaluation by introducing capability-conditioned navigation scenarios. The benchmark defines five representative agents: a wheelchair user, a person with a walker, an ambulatory person, a sweeping robot, and a quadruped robot. Each agent is characterized by specific attributes including physical dimensions (height, width, ground clearance), mobility capabilities (stair traversal, narrow passage navigation), and environmental interaction abilities.

The dataset construction involved capturing 45 diverse indoor scenes from real-world environments, ensuring ecological validity. Each scene contains multiple potential obstacles and navigation challenges that differentially affect agents based on their capabilities. For instance, a 15cm step poses no challenge to a quadruped robot but completely blocks a wheelchair user’s path.

The evaluation framework consists of two primary tasks: (1) navigation feasibility assessment, where models must determine whether a given path is traversable for a specific agent, and (2) obstacle identification and reasoning, where models must identify which obstacles prevent navigation and explain why. This dual-task structure tests both perceptual capabilities and spatial reasoning abilities.

Experimental Results and Analysis

The evaluation of 13 VLMs, including GPT-4V, Gemini, Claude, and various open-source models, reveals several critical findings. First, there exists a strong inverse correlation between mobility constraints and navigation accuracy. Models that achieve 75-80% accuracy for ambulatory agents drop to 45-55% for wheelchair users, indicating systematic failure in reasoning about restrictive mobility profiles.

Second, obstacle type significantly impacts performance. Models perform reasonably well on obvious barriers like walls and closed doors (65-70% accuracy) but struggle dramatically with dimensional obstacles requiring spatial reasoning. For obstacles like narrow doorways, low-hanging objects, or steps, accuracy drops to 35-45%, even for state-of-the-art models. This suggests VLMs lack robust mechanisms for comparing agent dimensions against environmental constraints.

Third, error analysis reveals that models frequently make two types of mistakes: (1) overgeneralization, where they assume all agents have similar capabilities, and (2) dimensional blindness, where they fail to reason about whether an agent’s physical dimensions allow passage through a space. Interestingly, providing explicit dimensional information in prompts improves performance by only 8-12%, suggesting the limitation is not merely about information availability but about spatial reasoning capabilities.

Implications for Embodied AI

The findings have significant implications for deploying VLMs in real-world navigation systems. Current models’ inability to reliably reason about capability constraints poses safety risks, particularly for assistive technologies serving users with mobility limitations. A navigation system that incorrectly suggests a route with stairs to a wheelchair user could lead to dangerous situations or significant inconvenience.

The research highlights the need for capability-aware training paradigms. Future VLM development should incorporate diverse agent perspectives during training, with explicit supervision on spatial reasoning tasks. This might involve synthetic data generation with varied agent profiles or curriculum learning approaches that progressively introduce more complex capability constraints.

Additionally, the benchmark reveals opportunities for architectural improvements. Models might benefit from dedicated spatial reasoning modules that explicitly compute dimensional compatibility, rather than relying solely on learned visual patterns. Hybrid approaches combining neural perception with symbolic spatial reasoning could potentially bridge the current performance gap.

Takeaways

1. Current VLMs show significant limitations in capability-conditioned navigation, with performance dropping 25-35% when evaluating agents with restrictive mobility constraints compared to ambulatory agents.

2. Dimensional reasoning represents a critical weakness—models struggle to assess whether agents can physically fit through spaces or traverse obstacles based on height, width, and clearance requirements.

3. The CapNav benchmark provides a standardized evaluation framework with 45 real-world scenes and 2365 QA pairs, enabling systematic assessment of capability-aware navigation abilities across diverse agent types.

4. Explicit capability information in prompts provides only marginal improvements (8-12%), suggesting the core limitation lies in spatial reasoning capabilities rather than information availability.

5. Future VLM development should prioritize capability-aware training paradigms and potentially incorporate dedicated spatial reasoning modules to enable safe deployment in real-world assistive navigation systems.