WHOLE: Holistic Hand-Object Reconstruction from Egocentric Videos

Paper: 2602.22209 Authors: Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu Categories: cs.CV

Abstract

Egocentric manipulation videos present unique challenges for 3D reconstruction due to severe occlusions during hand-object interactions and objects frequently moving in and out of the camera’s field of view. WHOLE addresses these challenges by introducing a holistic approach that jointly reconstructs hand and object motion in world space from egocentric videos. Unlike existing methods that process hands and objects independently—leading to inconsistent spatial relationships—WHOLE learns a generative prior over hand-object motion to reason about their interactions jointly. At inference time, this pretrained prior generates trajectories that align with video observations, substantially outperforming separate reconstruction approaches. The method achieves state-of-the-art results on hand motion estimation, 6D object pose estimation, and hand-object interaction reconstruction.

Key Contributions

• A joint generative framework that learns priors over hand-object motion to capture interaction patterns and spatial relationships
• World-space reconstruction that maintains consistency even when objects exit and re-enter the camera view
• A test-time optimization strategy that guides the generative prior to conform to video observations
• State-of-the-art performance on multiple benchmarks for hand tracking, object pose estimation, and interaction reconstruction

Technical Approach

The core innovation of WHOLE lies in its generative modeling of hand-object interactions. Rather than treating hand and object reconstruction as separate problems, the method learns a joint distribution over their motions in world space. This is achieved through a diffusion-based generative model trained on hand-object interaction data.

The architecture consists of two main components: a generative prior network and an observation guidance module. The prior network learns the manifold of plausible hand-object configurations and their temporal dynamics. During training, it captures patterns such as contact constraints, physical plausibility, and typical manipulation trajectories.

At test time, the method performs guided generation where the pretrained prior is conditioned on visual observations from the egocentric video. The guidance mechanism ensures that generated trajectories remain consistent with detected hand keypoints and object features while respecting the learned interaction constraints. This approach naturally handles occlusions and out-of-view scenarios by leveraging the learned prior to fill in missing information.

The world-space formulation is particularly important for egocentric videos where camera motion is significant. By reconstructing in a global coordinate frame rather than camera space, WHOLE maintains consistent object trajectories even as the viewpoint changes dramatically.

Results and Performance

WHOLE demonstrates substantial improvements over existing methods across multiple evaluation metrics. On hand motion estimation, the method reduces tracking error by leveraging object context—when hands are occluded by objects, the learned interaction prior provides strong cues about likely hand configurations.

For 6D object pose estimation, WHOLE achieves state-of-the-art accuracy by jointly reasoning about hand-object contact. This is particularly effective during manipulation phases where traditional object-only methods struggle due to occlusions. The joint modeling also enables the method to maintain accurate pose estimates when objects temporarily leave the field of view, using hand motion to infer object location.

The most significant gains appear in interaction reconstruction metrics that evaluate the consistency of hand-object spatial relationships. Methods that process hands and objects independently often produce physically implausible configurations with interpenetration or floating hands. WHOLE’s joint approach naturally avoids these artifacts by learning valid interaction patterns.

Qualitative results show that the method produces smooth, physically plausible trajectories even in challenging scenarios with rapid motion, severe occlusions, and frequent viewpoint changes. The world-space reconstruction enables coherent visualization of manipulation sequences from novel viewpoints.

Implications and Future Directions

WHOLE represents a significant step toward understanding human manipulation from egocentric video. The joint modeling approach has implications beyond reconstruction—the learned interaction priors could inform robotic manipulation, enable better action recognition, or support AR/VR applications that require accurate hand-object tracking.

The generative framework is particularly promising for handling the inherent ambiguity in egocentric videos. When visual evidence is limited due to occlusion or motion blur, the learned prior provides reasonable completions based on typical interaction patterns. This is more robust than purely discriminative approaches that may fail catastrophically when inputs are degraded.

Future work could extend this approach to handle multiple objects, deformable objects, or two-handed manipulation. The generative modeling framework is flexible enough to incorporate these extensions. Additionally, the method currently requires object templates—relaxing this requirement through category-level modeling would increase practical applicability.

The success of learning interaction priors also suggests opportunities for incorporating physics-based constraints more explicitly, potentially through differentiable simulation or hybrid learned-physics models.

Takeaways

1. Joint modeling of hand-object interactions through generative priors substantially outperforms independent reconstruction approaches
2. World-space formulation is crucial for maintaining consistency in egocentric videos with significant camera motion
3. Learned interaction priors enable robust reconstruction even during occlusions and out-of-view scenarios
4. The method achieves state-of-the-art performance on hand tracking, object pose estimation, and interaction reconstruction
5. Generative modeling provides a principled way to handle ambiguity and missing information in challenging egocentric scenarios