Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Paper: 2602.21186 Authors: Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang Categories: cs.CV

Abstract

Vision-Language Models (VLMs) excel at 2D visual understanding but struggle with 3D spatial reasoning—a fundamental component of spatial intelligence. This paper introduces Spa3R, a self-supervised framework that learns view-invariant spatial representations directly from unposed multi-view images without requiring explicit 3D modalities. Built on the Predictive Spatial Field Modeling (PSFM) paradigm, Spa3R synthesizes feature fields for unseen views conditioned on compact latent representations, enabling holistic 3D scene understanding. When integrated into VLMs through a lightweight adapter (Spa3-VLM), the system achieves state-of-the-art 58.6% accuracy on the challenging VSI-Bench 3D VQA benchmark, demonstrating that spatial intelligence can emerge from 2D vision alone.

Key Contributions

• Predictive Spatial Field Modeling (PSFM): A novel paradigm that learns to predict feature fields for arbitrary views from a unified spatial representation, internalizing coherent 3D scene understanding without explicit geometry
• Spa3R Framework: A self-supervised encoder that processes unposed multi-view images to generate view-invariant spatial representations, eliminating the need for camera poses or 3D supervision
• Spa3-VLM Integration: A lightweight adapter architecture that grounds existing VLMs in global spatial context, enabling effective 3D reasoning capabilities
• State-of-the-art Performance: Achieves 58.6% accuracy on VSI-Bench 3D VQA, significantly outperforming prior methods that rely on explicit 3D modalities or view-conditioned priors

Methodology: Predictive Spatial Field Modeling

The core innovation of Spa3R lies in its PSFM paradigm, which reframes 3D understanding as a predictive modeling task. Rather than explicitly reconstructing 3D geometry, the framework learns to synthesize dense feature fields $\mathcal{F}(\mathbf{v})$ for arbitrary target views $\mathbf{v}$ given a set of source images $\{I_1, I_2, ..., I_N\}$.

The architecture consists of three main components:

1. Spatial Encoder: Processes multi-view images to extract a compact latent representation $\mathbf{z} \in \mathbb{R}^{D}$ that encodes view-invariant spatial information about the scene
2. Field Predictor: Conditioned on $\mathbf{z}$ and target view parameters, generates dense feature fields that capture both geometric and semantic properties
3. Self-Supervised Training: Uses reconstruction objectives on held-out views to learn spatial coherence without requiring 3D annotations

The key insight is that by learning to predict features for unseen viewpoints, the model must internalize a holistic understanding of 3D structure. This approach sidesteps the ill-posed problem of reconstructing explicit geometry from sparse 2D observations, instead learning representations that are directly useful for downstream reasoning tasks.

Integration with Vision-Language Models

Spa3-VLM extends the pre-trained Spa3R encoder to enable 3D reasoning in VLMs through a minimal architectural modification. The integration strategy involves:

• Frozen Spa3R Encoder: The pre-trained spatial encoder remains fixed, preserving learned 3D understanding
• Lightweight Adapter: A small trainable module projects Spa3R’s spatial representations into the VLM’s feature space
• Global Spatial Context: Unlike view-conditioned approaches, Spa3R provides a unified representation of the entire scene, enabling reasoning about spatial relationships across multiple viewpoints

This design philosophy prioritizes scalability and modularity. By keeping the spatial encoder frozen and using minimal additional parameters, Spa3-VLM can be applied to various VLM architectures without extensive retraining. The global spatial context proves particularly valuable for tasks requiring understanding of object relationships, occlusions, and spatial configurations that span multiple views.

Experimental Results and Analysis

Evaluation on VSI-Bench, a challenging benchmark for 3D visual question answering, demonstrates the effectiveness of the PSFM approach:

• Overall Accuracy: Spa3-VLM achieves 58.6% on 3D VQA tasks, establishing a new state-of-the-art
• Comparison to Baselines: Significantly outperforms methods that use explicit 3D representations (point clouds, depth maps) or view-conditioned geometric priors
• Ablation Studies: Removing the global spatial context or replacing PSFM with alternative 3D encoding strategies results in substantial performance degradation
• Generalization: The learned representations transfer effectively across different scene types and question categories

The results validate the central hypothesis that spatial intelligence can emerge from 2D vision through appropriate self-supervised objectives. The PSFM paradigm’s ability to learn holistic scene understanding without explicit 3D supervision suggests a scalable path forward for spatial reasoning in VLMs.

Takeaways

1. Spatial intelligence in VLMs can emerge from 2D multi-view images alone, without requiring explicit 3D modalities or camera pose information
2. Predictive Spatial Field Modeling (PSFM) provides a scalable self-supervised paradigm for learning view-invariant spatial representations
3. Global spatial context significantly outperforms view-conditioned geometric priors for 3D reasoning tasks
4. Lightweight adapter architectures enable effective integration of spatial understanding into existing VLMs without extensive retraining
5. The 58.6% accuracy on VSI-Bench establishes a new state-of-the-art for 3D visual question answering, demonstrating practical viability of the approach