HERMES Framework Achieves 54.0% Boost in Mobile Dexterous Manipulation Success Rates

A groundbreaking robotics framework named HERMES, which stands for Human-to-Robot Embodied Learning From Multi-Source Motion Data for Mobile Dexterous Manipulation, has demonstrated a significant 54.0% improvement in mobile manipulation success rates in real-world scenarios. The project, led by researchers from Tsinghua University, Shanghai Qi Zhi Institute, and Peking University, leverages diverse human motion data and advanced sim-to-real transfer techniques to enable robots to perform complex bimanual dexterous tasks.

The HERMES framework was recently highlighted by RoboPapers, a platform dedicated to sharing cutting-edge robotics research. In a tweet, RoboPapers announced, "Full episode dropping soon! Geeking out with @fancy_yzc @still_wtm on HERMES: Human-to-Robot Embodied Learning From Multi-Source Motion Data for Mobile Dexterous Manipulation." The post also credited @micoolcho and @chris_j_paxton as co-hosts, indicating an upcoming detailed discussion or presentation of the work.

The core of HERMES lies in its ability to translate heterogeneous human hand motions into physically plausible robotic behaviors using a unified reinforcement learning approach. Key contributors to the research include Zhecheng Yuan and Tianming Wei, both associated with Tsinghua University and Shanghai Qi Zhi Institute, who were tagged in the RoboPapers tweet. The team also includes Langzhe Gu, Pu Hua, Tianhai Liang, Yuanpei Chen, and Huazhe Xu.

To bridge the challenging "sim2real gap," HERMES employs an end-to-end, depth image-based transfer method. This enables the robot to generalize learned skills to real-world environments more effectively. Furthermore, the framework integrates a closed-loop Perspective-n-Point (PnP) localization mechanism, which significantly enhances navigation accuracy and precision, crucial for successful mobile manipulation.

Experimental results showcased HERMES's superior performance across various complex tasks, including object handover, cleaning, and scanning. The framework achieved a remarkable 67.8% average success rate across six real-world bimanual dexterous manipulation tasks, outperforming raw depth baselines by 54.5%. This improvement underscores HERMES's capability to address both visual and dynamic discrepancies between simulated and real environments.

The project's success in robustly transforming diverse human motion data into robot-executable behaviors, coupled with its effective sim-to-real transfer and precise navigation capabilities, positions HERMES as a significant advancement in the field of mobile dexterous robotics. The upcoming full episode from RoboPapers is expected to provide further insights into this innovative research.