Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control

Paper: 2602.18422 Authors: Linxi Xie, Lisong C. Sun, Ashley Neall, Tong Wu, Shengqu Cai, Gordon Wetzstein Categories: cs.CV

Abstract

This paper introduces a human-centric video world model designed for extended reality (XR) applications that responds to fine-grained user motion. Unlike existing video generation models that rely on coarse control signals like text prompts or keyboard inputs, this system accepts tracked head pose and joint-level hand poses as conditioning inputs. The authors propose an effective conditioning mechanism for diffusion transformers that enables 3D head and hand control, facilitating realistic hand-object interactions. The system employs a teacher-student distillation approach, training a bidirectional video diffusion model and distilling it into a causal, real-time interactive system. Human subject evaluations demonstrate that this “generated reality” system achieves superior task performance and significantly higher perceived control compared to baseline methods.

Key Contributions

• Introduction of a human-centric video world model conditioned on both 3D head pose and joint-level hand poses for embodied XR interactions
• Evaluation and comparison of existing diffusion transformer conditioning strategies for fine-grained motion control
• A novel conditioning mechanism that effectively integrates 3D spatial information for head and hand tracking into video generation
• A teacher-student distillation framework that converts a bidirectional diffusion model into a causal, interactive system suitable for real-time applications
• Comprehensive human subject evaluation demonstrating improved task performance and perceived control in generated virtual environments

Technical Methodology

The core technical innovation lies in the conditioning strategy for diffusion transformers. Traditional video generation models struggle with fine-grained spatial control because they typically process control signals through text embeddings or simple concatenation. This work addresses the challenge by designing a conditioning mechanism specifically for 3D pose data.

The system architecture consists of two main components: a teacher model and a student model. The teacher is a bidirectional video diffusion model that can access both past and future frames, allowing it to generate high-quality, temporally coherent videos. This model is trained on egocentric video data paired with tracked head and hand poses, learning the relationship between human motion and the resulting visual observations.

The conditioning mechanism processes head pose (6-DOF: position and orientation) and hand poses (joint-level articulation for both hands) through specialized encoders that preserve spatial relationships. These encoded representations are then injected into the diffusion transformer at multiple layers, enabling the model to maintain consistent 3D geometry throughout the generation process.

For real-time interaction, the bidirectional teacher model is distilled into a causal student model that generates frames autoregressively, conditioned only on past frames and current pose inputs. This distillation process preserves the quality of the teacher while enabling the low-latency generation required for interactive XR applications.

Results and Evaluation

The evaluation methodology combines quantitative metrics with human subject studies to assess both technical performance and user experience. The authors conducted experiments where participants performed various tasks in the generated virtual environments, comparing the proposed system against relevant baselines including text-conditioned video generation and simpler pose-conditioning approaches.

Quantitative results show that the proposed conditioning mechanism achieves better temporal consistency and spatial accuracy in hand-object interactions compared to baseline methods. The system maintains stable geometry even during complex hand manipulations, a critical requirement for believable virtual interactions.

Human subject evaluations reveal two key findings. First, participants achieved significantly better task completion rates when using the full pose-conditioned system compared to coarser control methods. Second, and perhaps more importantly, users reported a substantially higher perceived level of control over their actions in the virtual environment. This subjective measure of agency is crucial for XR applications, as it directly impacts user immersion and comfort.

The system demonstrates particular strength in scenarios involving dexterous hand movements, such as grasping objects, manipulating tools, or performing fine motor tasks. The joint-level hand pose conditioning enables the model to generate realistic hand-object contact and occlusion patterns that would be difficult to achieve with coarser control signals.

Implications for Extended Reality

This work represents a significant step toward truly interactive virtual world generation for XR applications. Current XR systems typically rely on pre-authored content or simple procedural generation, limiting the diversity and adaptability of virtual environments. By enabling real-time generation of egocentric video conditioned on user motion, this approach opens new possibilities for dynamic, personalized XR experiences.

The human-centric design philosophy is particularly noteworthy. Rather than treating the user as an external observer, the system places human motion at the center of the generation process. This aligns naturally with the embodied nature of XR, where users expect their physical actions to have meaningful consequences in the virtual world.

Potential applications extend beyond entertainment to include training simulations, telepresence, and assistive technologies. For example, the system could generate realistic training scenarios that adapt to a trainee’s actions, or create shared virtual spaces where remote collaborators can interact with virtual objects using natural hand gestures.

However, challenges remain. Real-time generation of high-resolution video requires substantial computational resources, and the current system’s latency may still be perceptible in some scenarios. Additionally, the model’s ability to generalize to novel objects and environments depends on the diversity of training data, suggesting that further work on data efficiency and few-shot adaptation would be valuable.

Takeaways

1. Fine-grained pose conditioning (head + joint-level hands) enables significantly more controllable and interactive video generation compared to text or keyboard-based control
2. Specialized conditioning mechanisms for 3D spatial data are crucial for maintaining geometric consistency in generated egocentric video
3. Teacher-student distillation provides an effective path from high-quality bidirectional models to real-time causal generation systems
4. Human subject evaluation reveals that perceived control is as important as technical metrics for XR applications
5. This approach bridges the gap between passive video generation and interactive world simulation, opening new possibilities for embodied AI and XR experiences