H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

TL;DR

H-RDT leverages large-scale human manipulation videos with 3D hand poses to improve robotic manipulation, using a two-stage training process that enhances performance across various tasks and settings.

Contribution

The paper introduces H-RDT, a diffusion transformer model trained on human manipulation data to boost robotic manipulation capabilities, addressing data scarcity and embodiment diversity challenges.

Findings

H-RDT outperforms training from scratch and state-of-the-art methods.

Achieves 13.9% and 40.5% improvements in simulation and real-world tasks.

Effective in single-task, multitask, few-shot, and robustness scenarios.

Abstract

Imitation learning for robotic manipulation faces a fundamental challenge: the scarcity of large-scale, high-quality robot demonstration data. Recent robotic foundation models often pre-train on cross-embodiment robot datasets to increase data scale, while they face significant limitations as the diverse morphologies and action spaces across different robot embodiments make unified training challenging. In this paper, we present H-RDT (Human to Robotics Diffusion Transformer), a novel approach that leverages human manipulation data to enhance robot manipulation capabilities. Our key insight is that large-scale egocentric human manipulation videos with paired 3D hand pose annotations provide rich behavioral priors that capture natural manipulation strategies and can benefit robotic policy learning. We introduce a two-stage training paradigm: (1) pre-training on large-scale egocentric human manipulation data, and (2) cross-embodiment fine-tuning on robot-specific data with modular action encoders and decoders. Built on a diffusion transformer architecture with 2B parameters, H-RDT uses flow matching to model complex action distributions. Extensive evaluations encompassing both simulation and real-world experiments, single-task and multitask scenarios, as well as few-shot learning and robustness assessments, demonstrate that H-RDT outperforms training from scratch and existing state-of-the-art methods, including Pi0 and RDT, achieving significant improvements of 13.9% and 40.5% over training from scratch in simulation and real-world experiments, respectively. The results validate our core hypothesis that human manipulation data can serve as a powerful foundation for learning bimanual robotic manipulation policies.