HERO: Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation

Paper: 2602.16705 Authors: Runpei Dong, Ziyan Li, Xialin He, Saurabh Gupta Categories: cs.RO, cs.CV

Abstract

This paper introduces HERO (Humanoid End-effector control for Robot Open-vocabulary manipulation), a new approach to visual loco-manipulation with humanoid robots. The system addresses the fundamental challenge of enabling robots to manipulate arbitrary objects in unstructured environments by combining the generalization capabilities of large vision models with precise control from simulation-based training. The key innovation is a residual-aware end-effector tracking policy that integrates classical inverse kinematics, learned neural forward models, goal adjustment, and replanning mechanisms. This hybrid approach achieves a 3.2x reduction in end-effector tracking error compared to existing methods. The system demonstrates robust performance across diverse real-world settings, successfully manipulating everyday objects on surfaces ranging from 43cm to 92cm in height.

Key Contributions

• Residual-Aware EE Tracking Policy: A novel control architecture that combines inverse kinematics with learned neural forward models to achieve 3.2x improvement in end-effector tracking accuracy
• Hybrid Classical-ML Approach: Integration of classical robotics techniques (IK, replanning) with machine learning components for robust control performance
• Modular Loco-Manipulation System: A complete pipeline leveraging open-vocabulary vision models for scene understanding paired with accurate low-level control
• Real-World Validation: Demonstrated generalization across diverse environments (offices, coffee shops) and object categories without requiring extensive real-world training data

Technical Methodology

The HERO system architecture consists of three primary components working in concert. First, the perception module uses open-vocabulary large vision models to identify target objects and estimate their 3D poses from RGB-D inputs. This provides semantic understanding without requiring object-specific training.

The core innovation lies in the end-effector tracking policy. Traditional approaches suffer from accumulated errors in forward kinematics, especially for humanoid robots with complex kinematic chains. HERO addresses this through a residual formulation: instead of directly commanding joint positions, the policy outputs residual corrections $\Delta \mathbf{x}_{EE}$ to the desired end-effector pose. These residuals are converted to reference joint trajectories via inverse kinematics, then tracked by a learned policy.

The learned component includes a neural forward kinematics model trained on simulation data to predict actual end-effector positions given joint states. This model compensates for sim-to-real gaps and modeling errors. Additionally, the system implements goal adjustment mechanisms that modify target poses based on tracking performance, and replanning that generates new trajectories when tracking degrades beyond acceptable thresholds.

The locomotion component handles whole-body coordination, ensuring stable base positioning while the arms execute manipulation tasks. This modular design allows independent optimization of perception, control, and locomotion subsystems.

Experimental Results and Analysis

The evaluation demonstrates significant improvements over baseline approaches. In simulation experiments, HERO achieves end-effector tracking errors of approximately 2-3cm, compared to 6-10cm for standard learned policies without the residual formulation. This 3.2x improvement directly translates to higher manipulation success rates.

Real-world experiments span multiple environments and object categories. The system successfully manipulates objects including mugs, apples, bottles, and toys across surfaces at varying heights (43-92cm). Success rates exceed 85% for pick-and-place tasks in novel environments, demonstrating strong generalization from the open-vocabulary vision component.

Ablation studies reveal the importance of each system component. Removing the neural forward model increases tracking error by 40%, while eliminating goal adjustment reduces success rates by 25%. The replanning mechanism proves critical for recovering from perturbations and unexpected obstacles.

The modular design enables systematic debugging and improvement. When failures occur, the system logs indicate whether issues stem from perception (object detection/pose estimation), planning (trajectory generation), or control (tracking accuracy). This transparency facilitates targeted improvements.

Implications and Future Directions

HERO represents a significant step toward generalizable humanoid manipulation by demonstrating that simulation-trained policies can achieve precise control when properly designed. The residual formulation and hybrid classical-ML approach offer a template for other robotic control problems where sim-to-real transfer is challenging.

The reliance on open-vocabulary vision models provides natural language interfaces for task specification, enabling non-expert users to command robots through simple descriptions. This accessibility could accelerate deployment in domestic and service settings.

However, limitations remain. The system currently handles rigid objects with relatively simple geometries. Deformable objects, articulated items, and contact-rich manipulation remain challenging. The 43-92cm height range, while practical for many scenarios, excludes ground-level and overhead manipulation.

Future work could explore learning the residual policy directly from real-world data to further close the sim-to-real gap, incorporating tactile sensing for contact-rich tasks, and extending to bimanual coordination for more complex manipulation primitives.

Takeaways

1. Combining classical robotics (inverse kinematics) with learned components (neural forward models) achieves better performance than either approach alone for humanoid control
2. Residual formulations that output corrections rather than absolute commands significantly improve tracking accuracy by reducing accumulated errors
3. Open-vocabulary vision models enable strong generalization to novel objects and environments without extensive real-world training data
4. Modular system design facilitates systematic evaluation and debugging, allowing independent optimization of perception, planning, and control
5. Simulation-trained policies can achieve real-world manipulation success when augmented with appropriate bridging mechanisms like learned forward models and replanning