EgoForce: Absolute 3D Hand Pose from Egocentric Monocular Video via Forearm Guidance

Paper: 2605.12498 Authors: Christen Millerdurai, Shaoxiang Wang, Yaxu Xie, Vladislav Golyanik, Didier Stricker, Alain Pagani Categories: cs.CV, cs.GR

The Gap

Monocular RGB methods for egocentric hand pose estimation have hit two walls. First, they suffer from depth-scale ambiguity—you can’t tell if a hand is small and close or large and far from a single image. Second, head-mounted devices use wildly different optics (fisheye, perspective, distorted wide-FOV), and current models need expensive device-specific datasets for each configuration. Training on HoloLens data doesn’t transfer to Quest, which doesn’t transfer to Magic Leap.

Prior work either ignores absolute scale (outputs relative pose) or assumes a fixed camera model. Methods like HaMeR and HAMER-Ego predict hand pose but not its absolute position in camera space. Device-specific approaches like those trained on H2O or AssemblyHands achieve good accuracy but collapse when the camera changes.

Problem: Monocular egocentric hand pose
    |
    v
Limitation 1: Depth-scale ambiguity
    |                    +---> Relative pose only (HaMeR, HAMER-Ego)
    +---> Prior work ----+
    |                    +---> Device-specific training (H2O, AssemblyHands)
    v
Limitation 2: Camera model diversity
    |
    v
Gap: No unified method for absolute 3D pose across camera types
    |
    v
EgoForce: Forearm as scale anchor + ray space solver
    |
    v
Evidence: 28% MPJPE reduction on HOT3D, consistent across fisheye/perspective
    |
    v
Conclusion: Forearm geometry breaks scale ambiguity; ray space enables cross-device generalization

The Increment

One sentence: Before EgoForce, you needed separate models for each head-mounted device and got only relative hand pose; after EgoForce, a single model outputs absolute 3D hand position across fisheye, perspective, and wide-FOV cameras.

Core Mechanism

EgoForce has three components working in sequence. First, a differentiable forearm representation models the arm as a kinematic chain with known bone lengths (forearm is ~25cm in adults). This acts as a physical ruler in the scene. Second, a unified arm-hand transformer takes the egocentric image and predicts both hand joints and forearm geometry (elbow position, wrist orientation) in a shared latent space. The transformer is trained on data from multiple camera types simultaneously, learning camera-agnostic features. Third, a ray space closed-form solver takes the predicted 2D hand keypoints and forearm geometry, then solves for absolute 3D positions by intersecting rays from the camera with the forearm constraint.

Data flows like this: raw image → transformer → 2D keypoints + forearm parameters → ray space solver → absolute 3D hand pose in camera coordinates. The forearm acts as a scale reference—if you know the elbow-to-wrist distance is 25cm and you see both in the image, you can triangulate absolute depth. The ray space formulation handles different camera models (fisheye distortion, perspective projection) by working in a unified ray representation before projecting to 3D.

Input Image (fisheye/perspective/wide-FOV)
    |
    v
[Unified Arm-Hand Transformer]
    |
    +---> 2D hand keypoints (21 joints)
    |
    +---> Forearm geometry (elbow pos, wrist orient, bone lengths)
    |
    v
[Ray Space Solver]
    |  (intersect camera rays with forearm constraint)
    |
    v
Absolute 3D Hand Pose (camera space, metric scale)

Think of it like surveying land with a measuring tape. You’re standing on a hill (the camera) trying to map where someone’s hand is in 3D space. You can see the hand’s silhouette (2D keypoints), but you don’t know if it’s a child’s hand close by or an adult’s hand far away—that’s the scale ambiguity. Now imagine the person extends their arm, and you know their forearm is exactly 25cm long. That’s your measuring tape. You see where the elbow and wrist appear in your view, and because you know the real-world distance between them, you can calculate how far away the whole arm-hand system is. The “ray space solver” is like drawing sight lines from your eye through each joint you see, then finding where those lines must intersect in 3D to satisfy the constraint that the forearm is 25cm long. Different camera lenses (fisheye vs normal) bend those sight lines differently, but the measuring tape (forearm) stays the same length, so the math still works.

Key Concepts

• Depth-scale ambiguity: In a single 2D image, a small object close to the camera looks identical to a large object far away—same pixel size, same shape. Without stereo vision or other depth cues, you can’t tell which is true. For hands, this means you can estimate joint angles (relative pose) but not absolute distance from the camera. Prior monocular methods output hand pose in a normalized coordinate system (e.g., root joint at origin, hand scaled to unit size), which is useless for AR/VR applications that need to know “is the hand 30cm or 60cm from my face?” EgoForce breaks this by using the forearm as a known-length reference—like holding a ruler in the scene.

• Ray space representation: Different cameras project 3D points to 2D differently. A perspective camera uses linear projection (straight lines stay straight). A fisheye camera uses radial distortion (straight lines curve). Handling each requires different math. Ray space unifies this: instead of working with pixel coordinates (x, y), you work with the 3D ray direction from the camera center through each pixel. A 2D point (x, y) becomes a ray vector (dx, dy, dz). Now all camera models are just different ways to compute that ray direction. The solver works in ray space, so it doesn’t care whether the input was fisheye or perspective—it just sees rays and solves for 3D positions that project back onto those rays while satisfying the forearm constraint.

• Differentiable forearm model: The forearm is modeled as a kinematic chain (shoulder → elbow → wrist) with fixed bone lengths. “Differentiable” means you can compute gradients through this model during training. Why does this matter? The network predicts forearm parameters (joint angles, positions), and the solver uses those to compute 3D hand pose. If the final 3D pose is wrong, gradients flow backward through the solver into the forearm parameters, teaching the network to predict better forearm geometry. This end-to-end training is why the forearm constraint actually helps—it’s not just a post-processing step, it’s baked into the learning process.

Framework Shift

Before (mainstream approach):        After (EgoForce):

Camera Image                         Camera Image
    |                                    |
    v                                    v
[Hand Pose Network]                  [Arm-Hand Transformer]
    |                                    |
    +---> Relative 3D pose               +---> 2D keypoints
          (normalized, no scale)         +---> Forearm geometry
    |                                    |
    v                                    v
Output: Hand shape/joints            [Ray Space Solver]
        in arbitrary scale               |  (forearm = scale anchor)
                                         v
Train separately for each            Output: Absolute 3D pose
camera type (fisheye, perspective)           in camera space
                                         
                                     Train once, works across
                                     all camera types

One sentence: From predicting relative hand pose per camera type to predicting absolute hand position via forearm-anchored ray solving across unified camera models, the core shift is treating the arm as a metric ruler rather than ignoring it.

Expert Assessment

Problem choice: Real gap. AR/VR applications genuinely need absolute hand position (not just pose) for interaction, and the device fragmentation problem is acute—every headset uses different optics. The forearm insight is clever: it’s always visible in egocentric views and provides a known-length reference. Not manufactured.

Method maturity: The forearm constraint is the key insight; the rest is solid engineering. The ray space formulation is elegant—it’s the right abstraction for handling diverse cameras. However, the paper doesn’t deeply explore failure modes: what happens when the forearm is occluded or foreshortened? The differentiable forearm model is standard inverse kinematics, not novel. The transformer architecture is off-the-shelf (ViT backbone). The novelty is in the combination, not individual components.

Experimental integrity: Baselines are fair (HaMeR, HAMER-Ego, recent egocentric methods). The 28% MPJPE reduction on HOT3D is substantial. Cross-dataset evaluation (HOT3D, H2O, AssemblyHands) strengthens the generalization claim. One concern: the paper doesn’t report failure rates or provide error analysis by hand pose (e.g., does it fail on extreme wrist angles?). Ablations are present but could be more thorough—what’s the contribution of each component in isolation?

Writing quality: The method section is dense and assumes familiarity with ray tracing and inverse kinematics. The forearm representation could be explained more intuitively upfront. The related work section is thorough but reads like a literature dump. The results section is strong—clear tables, good visualizations. If I were rewriting, I’d lead with the forearm-as-ruler intuition in the introduction and defer mathematical details to an appendix.

Verdict: weak accept — Solves a real problem with a clever insight (forearm as scale anchor) and demonstrates strong empirical results, but the method is more engineering than algorithmic breakthrough, and the evaluation could probe failure modes more deeply.

Takeaways

Forearm as a scale anchor: If your task involves estimating metric 3D properties from monocular images, look for a known-size reference in the scene. For hands, it’s the forearm. For faces, it could be interpupillary distance. For full-body pose, it could be torso height. The key is finding something that’s (1) always visible, (2) has low variance across people, and (3) can be modeled differentiably.

Ray space for camera unification: If you’re building a vision system that needs to work across different camera models (fisheye, perspective, omnidirectional), don’t write separate code paths for each. Convert pixel coordinates to 3D ray directions as early as possible, do all processing in ray space, then project back to pixels only at the end. This is a general pattern for camera-agnostic algorithms.

End-to-end differentiable constraints: When you have domain knowledge (e.g., bone lengths are fixed), encode it as a differentiable module in your network rather than as a post-processing step. This lets gradients flow through the constraint during training, which is more effective than trying to satisfy the constraint after the fact. The forearm model here is a good example—it’s not just used for inference, it’s part of the loss function during training.