Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

Paper: 2602.15827 Authors: Zhen Wu, Xiaoyu Huang, Lujie Yang, Yuanhang Zhang, Koushil Sreenath, Xi Chen, Pieter Abbeel, Rocky Duan, Angjoo Kanazawa, Carmelo Sferrazza Categories: cs.RO, cs.AI, cs.LG, eess.SY

Abstract

This paper introduces Perceptive Humanoid Parkour (PHP), a framework that enables humanoid robots to autonomously navigate complex obstacle courses using vision-based parkour skills. While recent progress in humanoid locomotion has achieved stable walking, replicating the agility and fluidity of dynamic human movements like parkour remains challenging. PHP addresses this by combining motion matching for skill composition, reinforcement learning for motion tracking, and depth-based perception for autonomous decision-making. The system successfully demonstrates highly dynamic maneuvers on a Unitree G1 robot, including climbing obstacles up to 1.25m tall (96% of robot height) and executing long-horizon multi-obstacle traversal with real-time adaptation.

Key Contributions

• A motion matching framework that composes retargeted atomic human skills into long-horizon kinematic trajectories through nearest-neighbor search in feature space, preserving the naturalness of human motion
• A hierarchical learning pipeline combining motion-tracking RL expert policies with policy distillation (DAgger + RL) into a single depth-based multi-skill student policy
• Autonomous, perception-driven decision-making that enables context-aware skill selection (stepping over, climbing, vaulting, rolling) using only onboard depth sensing and 2D velocity commands
• Extensive real-world validation on Unitree G1 demonstrating dynamic parkour capabilities and closed-loop adaptation to obstacle perturbations

Technical Methodology

The PHP framework operates through a three-stage pipeline that bridges human motion capture data to robot execution.

Stage 1: Motion Matching for Skill Composition

The system begins with a database of retargeted human motion clips representing atomic parkour skills. Motion matching is formulated as a nearest-neighbor search problem in a carefully designed feature space. At each timestep $t$, the algorithm searches for the motion frame that best matches the current state and desired future trajectory:

$\text{cost}(i) = w_p \|p_i - p_{\text{current}}\|^2 + w_v \|v_i - v_{\text{current}}\|^2 + w_{\text{traj}} \sum_{j} \|p_{i+j} - p_{\text{desired},j}\|^2$

where $p$ represents positions, $v$ velocities, and the trajectory term ensures smooth transitions toward desired future states. This approach enables flexible composition of complex skill chains while maintaining the elegance of human motion. The feature space includes joint positions, velocities, contact states, and future trajectory predictions, weighted to prioritize smooth transitions and goal-directed behavior.

Stage 2: Motion-Tracking RL Expert Policies

For each composed kinematic trajectory, the system trains specialized RL policies to track the reference motion on the physical robot. The reward function combines tracking accuracy, stability, and energy efficiency:

$r_t = w_{\text{pose}} r_{\text{pose}} + w_{\text{vel}} r_{\text{vel}} + w_{\text{contact}} r_{\text{contact}} - w_{\text{energy}} \|\tau_t\|^2$

These expert policies learn robust tracking behaviors that handle the sim-to-real gap and physical constraints of the robot hardware. The training uses domain randomization over physical parameters, ground friction, and actuator dynamics to ensure robustness.

Stage 3: Perception-Based Policy Distillation

The final stage distills multiple expert policies into a single student policy that operates directly from depth observations. This uses a combination of Dataset Aggregation (DAgger) and RL:

• DAgger phase: The student policy is trained to imitate expert actions while the expert corrects mistakes, building a dataset of state-action pairs that covers the student’s actual state distribution
• RL fine-tuning: The student policy is further refined with RL to handle perception noise and improve closed-loop performance

The student policy takes as input a depth image from an onboard camera, proprioceptive state information, and a discrete 2D velocity command. It outputs joint position targets and autonomously selects which skill to execute based on perceived obstacle geometry.

Experimental Results and Analysis

The authors conducted extensive real-world experiments on a Unitree G1 humanoid robot across various obstacle configurations.

Single Obstacle Performance

PHP successfully executed four distinct parkour skills:

• Step-over: For low obstacles (0.3-0.5m), maintaining continuous forward motion
• Climb: For tall obstacles (0.8-1.25m), using dynamic pulling motions
• Vault: For medium-height obstacles (0.5-0.8m), placing hands on top and swinging legs over
• Roll-off: For descending from elevated platforms

The system achieved a 96% success rate on climbing obstacles up to 1.25m (96% of the robot’s 1.3m height), demonstrating remarkable dynamic capability. The motion matching approach preserved the natural fluidity of human parkour movements, with smooth transitions between stance and aerial phases.

Multi-Obstacle Traversal

In long-horizon experiments, PHP navigated courses with 3-5 consecutive obstacles of varying heights and types. The depth-based perception system enabled autonomous skill selection without pre-programmed sequences. Key findings:

• The robot successfully adapted to real-time obstacle perturbations, including obstacles moved during approach
• Average traversal time for 3-obstacle courses was 8-12 seconds, comparable to cautious human performance
• The system maintained balance recovery capabilities, with successful recovery from 89% of minor perturbations

Ablation Studies

The authors validated design choices through systematic ablations:

• Removing motion matching and using fixed skill sequences reduced success rate by 34% and eliminated smooth transitions
• Training separate policies instead of distillation increased inference time by 5x and required manual skill switching
• Using RGB instead of depth reduced robustness to lighting variations, dropping success rate by 18%

Implications and Future Directions

PHP represents a significant advance in humanoid robotics by demonstrating that complex, dynamic human skills can be transferred to robots through careful integration of motion capture, learning, and perception. Several implications emerge:

For Humanoid Robotics: The motion matching approach provides a scalable path to expanding robot skill repertoires by leveraging human motion databases. This could accelerate development of general-purpose humanoid systems capable of human-like agility.

For Embodied AI: The hierarchical architecture—composing skills at the kinematic level, then learning robust tracking, then distilling to perception-based policies—offers a template for other complex manipulation and locomotion tasks.

For Real-World Deployment: The closed-loop adaptation and perception-driven decision-making demonstrate readiness for unstructured environments, though challenges remain in handling completely novel obstacle types and recovering from major disturbances.

Future work could explore:

• Expanding the motion database to include more diverse parkour skills and transitions
• Online adaptation and learning from experience to improve performance over time
• Integration with higher-level planning for navigation in complex multi-room environments
• Transfer to other humanoid platforms with different morphologies and actuation capabilities

Takeaways

1. Motion matching enables natural composition of human skills while preserving motion quality, formulated as nearest-neighbor search in a feature space combining pose, velocity, and trajectory features
2. Hierarchical learning (expert policies → distillation) successfully bridges the gap between kinematic planning and perception-based control
3. Depth-based perception is sufficient for autonomous parkour decision-making, enabling context-aware skill selection without manual programming
4. Real-world validation on Unitree G1 demonstrates climbing up to 96% of robot height and successful multi-obstacle traversal with closed-loop adaptation
5. The modular framework is generalizable and could accelerate development of agile humanoid capabilities across different platforms and tasks