PanoEnv: Teaching Vision-Language Models 3D Spatial Reasoning in Panoramic Environments

Paper: 2602.21992 Authors: Zekai Lin, Xu Zheng Categories: cs.CV

Abstract

This paper addresses a critical limitation in current Vision-Language Models (VLMs): their inability to perform robust 3D spatial reasoning on 360° panoramic images. The authors introduce PanoEnv, a comprehensive benchmark containing 14.8K questions across five spatial reasoning categories, built from synthetic 3D environments with accurate depth, segmentation, and bounding box annotations. Evaluation of 14 state-of-the-art VLMs reveals significant deficiencies, with only 49.34% overall accuracy and a mere 8.36% on open-ended questions. To address this gap, the researchers propose a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) with geometry-aware reward strategies and a two-stage curriculum that mitigates catastrophic forgetting. Their 7B parameter model achieves 52.93% overall accuracy and 14.83% on open-ended questions, surpassing even 32B models in semantic evaluation metrics.

Key Contributions

• PanoEnv-QA Benchmark: A large-scale dataset with 14.8K questions spanning five 3D spatial reasoning categories (relative position, distance estimation, volume comparison, spatial relationship, and scene understanding) grounded in accurate 3D annotations from synthetic environments
• Comprehensive VLM Evaluation: Systematic benchmarking of 14 state-of-the-art VLMs revealing significant limitations in 3D spatial reasoning on Equirectangular Projection (ERP) images
• GRPO-based RL Framework: A reinforcement learning post-training approach incorporating five geometry-aware reward strategies (distance tolerance, spatial consistency, multi-object reasoning, scale awareness, and viewpoint invariance)
• Two-Stage Curriculum Learning: A training strategy that first focuses on structured tasks (true/false and multiple choice) before introducing open-ended questions, effectively preventing catastrophic forgetting
• State-of-the-Art Performance: A 7B model that achieves superior results compared to larger models, with Q-Score of 6.24 and P-Score of 5.95

Technical Methodology

The PanoEnv framework operates on multiple technical levels to address 3D spatial reasoning challenges. The benchmark construction leverages synthetic 3D environments to generate panoramic images with pixel-perfect ground truth annotations, eliminating the ambiguity inherent in real-world data collection. The Equirectangular Projection format preserves the full 360° field of view while introducing geometric distortions that challenge standard computer vision approaches.

The reinforcement learning framework builds upon Group Relative Policy Optimization (GRPO), which offers advantages over traditional policy gradient methods by computing relative advantages within groups of responses. The reward function $R(q, a, \hat{a})$ incorporates five geometry-aware strategies:

$R_{total} = \alpha_1 R_{dist} + \alpha_2 R_{spatial} + \alpha_3 R_{multi} + \alpha_4 R_{scale} + \alpha_5 R_{view}$

where each component evaluates different aspects of spatial reasoning. The distance tolerance reward $R_{dist}$ allows for acceptable margins in numerical predictions, while spatial consistency $R_{spatial}$ ensures logical coherence across related questions. Multi-object reasoning $R_{multi}$ rewards correct handling of complex scenes with multiple entities, scale awareness $R_{scale}$ penalizes magnitude errors in size estimation, and viewpoint invariance $R_{view}$ encourages consistent reasoning across different panoramic perspectives.

The two-stage curriculum addresses the challenge of catastrophic forgetting when transitioning from structured to open-ended tasks. Stage 1 trains exclusively on true/false and multiple-choice questions where the model learns fundamental spatial concepts with clear feedback signals. Stage 2 introduces a mixed dataset combining structured and open-ended questions with carefully tuned sampling ratios, allowing the model to generalize while maintaining previously acquired knowledge.

Experimental Results and Analysis

The benchmark evaluation reveals striking performance gaps across model architectures. Proprietary models like GPT-4V and Claude achieve 52-55% accuracy on structured tasks but drop to 10-12% on open-ended questions. Open-source models show even larger disparities, with LLaVA-1.5 achieving 45% on multiple choice but only 6% on open-ended spatial reasoning. This pattern indicates that current VLMs rely heavily on pattern matching rather than genuine 3D understanding.

The proposed RL framework demonstrates consistent improvements across all question categories. On relative position questions, accuracy increases from 51.2% to 54.8%. Distance estimation sees gains from 46.7% to 50.1%, while volume comparison improves from 48.9% to 52.3%. Most notably, open-ended question accuracy nearly doubles from 8.36% to 14.83%, suggesting that the geometry-aware rewards successfully instill spatial reasoning capabilities.

Ablation studies reveal the importance of each curriculum component. Removing Stage 1 structured training reduces overall accuracy by 4.2%, while eliminating the mixed-data Stage 2 causes a 6.7% drop in open-ended performance. The geometry-aware reward components contribute differentially: spatial consistency provides the largest individual gain (+2.1%), followed by distance tolerance (+1.8%) and multi-object reasoning (+1.5%).

The semantic evaluation metrics (Q-Score and P-Score) provide additional validation. The 7B model’s Q-Score of 6.24 surpasses GPT-4V’s 6.01 and significantly exceeds the 32B baseline’s 5.87. This demonstrates that the RL framework enables smaller models to achieve superior semantic understanding through targeted spatial reasoning training rather than simply scaling parameters.

Implications and Future Directions

PanoEnv establishes a new paradigm for evaluating and enhancing 3D spatial intelligence in vision-language models. The benchmark’s synthetic foundation ensures reproducibility and enables controlled experimentation, while the diversity of question types captures the multifaceted nature of spatial reasoning. The significant performance gaps revealed by the evaluation suggest that current VLM architectures may require fundamental modifications to handle geometric reasoning effectively.

The success of the GRPO-based framework indicates that reinforcement learning with carefully designed reward functions can instill capabilities that emerge weakly from standard supervised training. The geometry-aware strategies provide explicit signals for spatial concepts that are difficult to capture through next-token prediction alone. This approach could extend to other domains requiring structured reasoning, such as temporal understanding or causal inference.

The curriculum learning strategy offers insights into efficient knowledge transfer. By separating structured and open-ended training phases, the framework avoids the instability that often accompanies multi-task learning. The mixed-data Stage 2 acts as a bridge, allowing the model to generalize while maintaining foundational knowledge. This principle could inform training strategies for other complex reasoning tasks.

Future work could explore several promising directions: extending the benchmark to real-world panoramic images with noisy annotations, investigating architectural modifications specifically designed for ERP geometry, incorporating multi-modal signals like depth sensors or LiDAR, and scaling the RL framework to larger models. The framework’s success with a 7B model suggests that efficient training strategies may be more impactful than raw parameter scaling for specialized reasoning tasks.

Takeaways

1. Current state-of-the-art VLMs demonstrate significant deficiencies in 3D spatial reasoning on panoramic images, achieving only 49.34% overall accuracy and 8.36% on open-ended questions despite strong performance on standard vision-language benchmarks
2. PanoEnv provides a rigorous evaluation framework with 14.8K questions grounded in accurate 3D annotations, enabling systematic assessment of spatial reasoning capabilities across five distinct categories
3. Reinforcement learning with geometry-aware rewards can effectively instill 3D spatial intelligence, with the GRPO-based framework achieving +3.59% overall improvement and nearly doubling open-ended question accuracy
4. Two-stage curriculum learning successfully mitigates catastrophic forgetting, allowing models to master structured spatial reasoning before generalizing to open-ended tasks
5. Targeted training strategies can enable smaller models (7B parameters) to surpass larger models (32B parameters) on specialized reasoning tasks, achieving superior semantic evaluation scores (Q-Score 6.24 vs 5.87)
6. The five geometry-aware reward strategies (distance tolerance, spatial consistency, multi-object reasoning, scale awareness, viewpoint invariance) contribute complementarily to spatial reasoning performance
7. Synthetic 3D environments provide a scalable foundation for generating high-quality training data with pixel-perfect annotations, addressing the supervision gap in panoramic image understanding