VBVR: A Million-Scale Dataset for Video Reasoning at Scale

Paper: 2602.20159 Authors: Maijunxian Wang, Ruisi Wang, Juyi Lin, Ran Ji, Thaddäus Wiedemer, Qingying Gao, Dezhi Luo, Yaoyao Qian, Lianyu Huang, Zelong Hong Categories: cs.CV, cs.AI, cs.LG, cs.MM, cs.RO

Abstract

While video generation models have made impressive strides in visual quality, their reasoning capabilities remain largely unexplored. This paper addresses a critical gap in video understanding research by introducing the Very Big Video Reasoning (VBVR) Dataset, containing over one million video clips spanning 200 carefully curated reasoning tasks. The dataset is approximately three orders of magnitude larger than existing video reasoning benchmarks, enabling systematic study of scaling behavior in video reasoning models. Alongside the dataset, the authors present VBVR-Bench, a verifiable evaluation framework that uses rule-based, human-aligned scorers instead of model-based judging, ensuring reproducible and interpretable assessment. Through large-scale experiments, the research reveals early signs of emergent generalization to unseen reasoning tasks, suggesting that video reasoning capabilities may scale with data and model size in ways similar to language models.

Key Contributions

• Unprecedented Scale: VBVR contains over 1 million video clips across 200 reasoning tasks, roughly 1000× larger than existing video reasoning datasets
• Principled Taxonomy: A structured categorization of video reasoning tasks covering spatiotemporal continuity, object interactions, causal relationships, and physical dynamics
• Verifiable Evaluation: VBVR-Bench introduces rule-based scoring mechanisms that eliminate the subjectivity and inconsistency of model-based evaluation
• Scaling Study: First large-scale empirical investigation of how video reasoning capabilities emerge and generalize with increased data and model capacity
• Open Resources: Complete dataset, benchmark toolkit, and trained models publicly released at https://video-reason.com/

The Video Reasoning Gap

Current video models excel at generating visually coherent content but struggle with fundamental reasoning tasks that require understanding spatiotemporal structure. Unlike text, video provides grounded visual environments where reasoning must account for:

• Temporal Continuity: Understanding how scenes evolve consistently over time
• Spatial Relationships: Tracking object positions, orientations, and interactions in 3D space
• Causal Dynamics: Inferring cause-and-effect relationships from observed physical events
• Counterfactual Reasoning: Predicting alternative outcomes under different conditions

The lack of large-scale training data has prevented systematic study of these capabilities. Existing video reasoning datasets typically contain hundreds to thousands of examples, insufficient for training modern deep learning models or studying scaling laws. VBVR bridges this gap by providing a resource comparable in scale to language model pretraining datasets.

Dataset Design and Taxonomy

VBVR organizes its 200 reasoning tasks into a hierarchical taxonomy based on cognitive complexity and reasoning type:

Physical Reasoning: Tasks involving understanding of physics, including object dynamics, collision prediction, stability assessment, and force inference. These tasks test whether models grasp fundamental physical laws governing motion and interaction.

Temporal Reasoning: Tasks requiring understanding of event sequences, temporal ordering, duration estimation, and change detection. Models must track how states evolve and identify temporal dependencies.

Spatial Reasoning: Tasks focused on 3D understanding, including depth perception, occlusion reasoning, viewpoint transformation, and spatial relationship inference.

Causal Reasoning: Tasks that require identifying cause-effect relationships, predicting consequences, and understanding intervention effects in dynamic scenes.

Compositional Reasoning: Complex tasks combining multiple reasoning types, testing the model’s ability to integrate different cognitive capabilities.

Each task includes multiple difficulty levels, enabling fine-grained analysis of model capabilities. The dataset construction process involved careful curation to ensure diversity in visual content, scene complexity, and reasoning requirements.

VBVR-Bench: Verifiable Evaluation Framework

A critical innovation is the evaluation methodology. Traditional video understanding benchmarks often rely on model-based judges (e.g., using GPT-4 to score responses), which introduces several problems:

• Inconsistency: Model judges can produce different scores for identical inputs across runs
• Bias: Judges may favor certain response styles or exhibit systematic preferences
• Opacity: The scoring rationale is often unclear, making debugging difficult
• Cost: API-based evaluation can be expensive at scale

VBVR-Bench addresses these issues through rule-based, programmatic scorers aligned with human judgment. For each task, the framework defines explicit success criteria that can be automatically verified. This approach provides:

• Reproducibility: Identical inputs always produce identical scores
• Interpretability: Clear explanation of why a response succeeded or failed
• Efficiency: Fast evaluation without API calls or model inference
• Reliability: No dependence on external services or model availability

The human alignment process involved extensive validation to ensure rule-based scores correlate strongly with expert human judgments across diverse reasoning tasks.

Scaling Behavior and Emergent Generalization

Using VBVR, the authors conducted extensive scaling experiments varying:

• Data Scale: Training on subsets ranging from $10^3$ to $10^6$ examples
• Model Size: Testing architectures from small (millions of parameters) to large (billions of parameters)
• Task Diversity: Evaluating on held-out reasoning tasks not seen during training

Key findings include:

Power Law Scaling: Performance on video reasoning tasks follows power law relationships with both data size and model capacity, similar to language model scaling laws. This suggests that continued scaling may yield substantial improvements.

Emergent Generalization: Models trained on diverse reasoning tasks show non-trivial transfer to completely unseen task types. This zero-shot generalization emerges primarily in larger models trained on more diverse data, indicating that video reasoning may exhibit emergent capabilities analogous to those observed in large language models.

Task Composition: Models demonstrate ability to solve complex compositional tasks by combining simpler reasoning primitives learned during training, suggesting hierarchical learning of reasoning skills.

Bottlenecks: Physical reasoning tasks involving precise dynamics prediction remain challenging even for the largest models, indicating areas where current architectures may need architectural innovations beyond pure scaling.

Implications for Video AI

VBVR establishes video reasoning as a distinct research direction with unique challenges and opportunities:

Beyond Visual Quality: The field must move beyond perceptual metrics to evaluate genuine understanding of spatiotemporal structure and causal relationships.

Grounded Intelligence: Video provides a natural testbed for grounded reasoning that connects perception to physical understanding, potentially bridging vision and robotics.

Scaling Paradigm: The observed scaling behavior suggests that video reasoning may benefit from the same data-driven scaling approach that transformed language models, but requires domain-specific datasets like VBVR.

Evaluation Rigor: The success of rule-based evaluation demonstrates the importance of verifiable, interpretable benchmarks in AI research, especially as models become more capable.

Takeaways

1. VBVR provides the first dataset at sufficient scale (1M+ videos, 200 tasks) to enable systematic study of video reasoning and its scaling properties
2. Video reasoning exhibits early signs of emergent generalization to unseen tasks, suggesting that scaling laws may apply beyond language models
3. Rule-based evaluation frameworks can provide more reliable and interpretable assessment than model-based judges for structured reasoning tasks
4. Physical reasoning remains a significant challenge even for large models, indicating opportunities for architectural innovation
5. The principled taxonomy and open resources establish a foundation for the next generation of video understanding research focused on reasoning rather than just perception