Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation

Paper: 2602.15828 Authors: Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, Shubham Tulsiani Categories: cs.RO, cs.CV, cs.LG

Abstract

Dex4D addresses the fundamental challenge of learning generalist policies for dexterous manipulation without requiring expensive real-world data collection or task-specific reward engineering. The framework learns a domain-agnostic 3D point track conditioned policy in simulation that can manipulate any object to any desired pose. By training across thousands of objects with diverse pose configurations, Dex4D covers a broad space of robot-object interactions that can be flexibly composed at test time. The policy achieves zero-shot transfer to real-world tasks by being prompted with object-centric point tracks extracted from generated videos, and uses online point tracking for closed-loop control during execution.

Key Contributions

• A task-agnostic “Anypose-to-Anypose” policy that learns dexterous manipulation skills across thousands of simulated objects without task-specific rewards
• Zero-shot sim-to-real transfer capability enabled by 3D point track conditioning, eliminating the need for real-world finetuning
• A flexible prompting mechanism using point tracks extracted from generated videos to specify desired manipulation behaviors
• Closed-loop perception and control through online point tracking during policy execution
• Demonstrated strong generalization to novel objects, scene layouts, backgrounds, and trajectories in both simulation and real-world experiments

Methodology

The Dex4D framework consists of three key components that work together to enable task-agnostic dexterous manipulation.

Training in Simulation: The policy is trained in simulation using a diverse dataset of thousands of objects with varied pose configurations. Rather than designing task-specific environments and rewards, Dex4D learns to manipulate objects based on 3D point track conditions. This “Anypose-to-Anypose” formulation allows the policy to learn a broad distribution of robot-object interactions that can be recomposed at test time. The training process covers a wide range of manipulation primitives including grasping, reorienting, placing, and fine-grained adjustments.

Point Track Conditioning: The core innovation is conditioning the policy on 3D point tracks rather than high-level task specifications or demonstrations. Point tracks provide a natural, object-centric representation that is both expressive enough to specify complex manipulation behaviors and general enough to transfer across domains. At deployment, desired point tracks can be extracted from generated videos showing the intended manipulation, providing an intuitive interface for specifying tasks without manual programming.

Closed-Loop Execution: During real-world execution, Dex4D employs online point tracking to maintain closed-loop perception and control. The system continuously tracks the 3D positions of key points on the object and adjusts the robot’s actions accordingly. This feedback mechanism enables the policy to handle perturbations, uncertainties, and domain shifts that naturally occur when transferring from simulation to reality.

Results and Evaluation

Extensive experiments validate Dex4D’s effectiveness across multiple dimensions. In simulation, the method demonstrates superior performance compared to prior baselines on a diverse set of manipulation tasks. The task-agnostic nature of the learned policy allows it to handle novel task compositions without additional training.

Real-world experiments showcase the zero-shot transfer capability of Dex4D. Without any finetuning on real data, the policy successfully performs various dexterous manipulation tasks including object reorientation, precise placement, and multi-step manipulations. The system exhibits robust generalization to objects not seen during training, different scene configurations, varied backgrounds, and novel manipulation trajectories.

Ablation studies reveal the importance of key design choices. The 3D point track conditioning proves crucial for enabling sim-to-real transfer, outperforming alternative representations. The diversity of training data in simulation directly correlates with generalization performance in the real world. Online point tracking during execution significantly improves success rates compared to open-loop control.

Implications and Future Directions

Dex4D represents a significant step toward scalable learning of generalist manipulation policies. By decoupling skill learning from task specification, the framework addresses two major bottlenecks in dexterous manipulation: the cost of real-world data collection and the engineering effort required for task-specific reward design.

The point track conditioning mechanism offers a promising interface for human-robot interaction. Users can specify desired manipulations through intuitive visual demonstrations or generated videos, without needing to understand low-level robot control or write explicit programs. This could democratize access to robotic manipulation capabilities.

Several directions remain for future work. Extending the framework to handle deformable objects and contact-rich manipulations would broaden its applicability. Incorporating multi-modal sensing beyond vision could improve robustness in challenging conditions. Exploring hierarchical composition of learned skills could enable even more complex, long-horizon manipulation tasks. Finally, investigating how to efficiently adapt the policy to new robot morphologies would enhance its practical deployment potential.

Takeaways

1. Task-agnostic learning in simulation can produce policies that zero-shot transfer to diverse real-world dexterous manipulation tasks
2. 3D point track conditioning provides an effective interface between high-level task specification and low-level robot control
3. Training across thousands of objects with diverse configurations enables strong generalization to novel scenarios
4. Online point tracking during execution is crucial for robust closed-loop control in real-world settings
5. The framework eliminates the need for expensive real-world data collection and task-specific reward engineering
6. Generated videos can serve as an intuitive medium for specifying desired manipulation behaviors through point track extraction