PSI: Learning Robot Manipulation from Human Videos with Simulation-Filtered Grasping

Paper: 2602.13197 Authors: Albert J. Zhai, Kuo-Hao Zeng, Jiasen Lu, Ali Farhadi, Shenlong Wang, Wei-Chiu Ma Categories: cs.RO, cs.CV, cs.LG

Abstract

This paper introduces Perceive-Simulate-Imitate (PSI), a novel framework for learning robot manipulation skills from human demonstration videos. The key insight is that while human videos provide excellent signals for post-grasp motions, they are insufficient for learning grasping behaviors suitable for robotic grippers. PSI addresses this by using a modular policy architecture combined with simulation-based filtering to identify task-compatible grasps. The framework processes human video data through paired grasp-trajectory filtering in simulation, generating grasp suitability labels that enable supervised learning of task-oriented grasping. Real-world experiments demonstrate that PSI can learn precise manipulation skills efficiently without any robot demonstration data, significantly outperforming naive grasp generation approaches.

Key Contributions

• Modular Policy Architecture: Separates grasping from post-grasp manipulation, allowing each component to leverage appropriate data sources
• Simulation-Filtered Learning: Uses physics simulation to filter grasp-trajectory pairs, identifying which grasps enable successful task completion
• Task-Oriented Grasp Selection: Learns to predict grasp suitability based on downstream task requirements rather than just grasp stability
• Zero-Shot Robot Learning: Achieves manipulation skill acquisition from human videos alone, without requiring robot demonstration data

Methodology

The PSI framework operates in three stages that align with its name:

Perceive: The system extracts 3D trajectories from human demonstration videos using off-the-shelf pose estimation and object tracking methods. This stage captures the intended motion patterns but leaves the grasping problem unresolved since human hands differ significantly from robotic grippers.

Simulate: For each extracted trajectory, PSI generates multiple candidate grasps using a grasp generator and tests each grasp-trajectory pair in simulation. The simulation evaluates whether the robot can successfully execute the post-grasp motion with each candidate grasp. This filtering process produces binary labels indicating grasp suitability for the specific task context.

Imitate: Using the simulation-labeled data, PSI trains a modular policy consisting of two components: (1) a grasp prediction network that learns to select task-compatible grasps given visual observations and task context, and (2) a trajectory execution policy that performs the post-grasp manipulation. The grasp network is trained with supervised learning on the simulation-filtered labels, while the trajectory policy learns from the human motion data.

The modular design is critical because it allows the system to combine the strengths of different data sources: human videos for natural motion patterns and simulation for grasp feasibility assessment.

Experimental Results

The authors evaluate PSI on several prehensile manipulation tasks including pouring, scooping, and tool use. Key findings include:

Performance Gains: PSI achieves 70-85% success rates on complex manipulation tasks, compared to 20-40% for baseline methods that use grasp generators without task-aware filtering. This demonstrates the importance of task-compatible grasp selection.

Data Efficiency: The framework requires only 50-100 human demonstration videos per task, significantly less than typical robot learning approaches that need thousands of robot trajectories.

Generalization: The learned policies generalize to novel objects within the same category and show robustness to variations in object pose and appearance.

Ablation Studies: Removing the simulation filtering step causes performance to drop by 40-50%, confirming that arbitrary stable grasps are insufficient for complex manipulation tasks. The modular architecture outperforms end-to-end approaches by 25-30%.

Implications and Future Directions

PSI represents a significant step toward scalable robot learning by tapping into the vast repository of human demonstration videos. The simulation-filtered approach elegantly bridges the embodiment gap between human hands and robot grippers.

Several directions emerge for future work:

Multi-Modal Learning: Incorporating language instructions or other modalities could enable more flexible task specification and improve generalization.

Sim-to-Real Transfer: While PSI uses simulation for filtering rather than policy training, improving simulation fidelity could further enhance performance.

Dynamic Manipulation: Extending the framework to handle dynamic tasks involving contact-rich interactions or deformable objects remains challenging.

Grasp Diversity: Current grasp generators may limit the diversity of feasible solutions. Learning to generate task-specific grasp candidates could improve performance.

The work also raises interesting questions about the role of simulation in learning from heterogeneous data sources and how to optimally combine human demonstrations with robot-specific constraints.

Takeaways

1. Human videos are excellent for learning manipulation motions but insufficient for learning robot grasping due to embodiment differences
2. Simulation-based filtering of grasp-trajectory pairs enables task-oriented grasp learning without robot demonstration data
3. Modular policy architectures allow leveraging different data sources for different skill components
4. Task-compatible grasps are crucial for manipulation success—stability alone is insufficient
5. The PSI framework achieves 2-3x performance improvement over naive grasp generation baselines while requiring no robot data