DexH2R: Task-oriented Dexterous Manipulation from Human to Robots

TL;DR

DexH2R is a novel framework that enhances robotic dexterous manipulation by combining human hand motion retargeting with a residual policy, enabling high generalization and reducing data collection efforts.

Contribution

It introduces a task-oriented residual policy learned from retargeted actions, eliminating the need for complex teleoperation and improving generalization to new scenarios.

Findings

Outperforms prior methods by 40% in various tasks

Effective in both simulation and real-world environments

Enables learning new skills with high generalizability

Abstract

Dexterous manipulation is a critical aspect of human capability, enabling interaction with a wide variety of objects. Recent advancements in learning from human demonstrations and teleoperation have enabled progress for robots in such ability. However, these approaches either require complex data collection such as costly human effort for eye-robot contact, or suffer from poor generalization when faced with novel scenarios. To solve both challenges, we propose a framework, DexH2R, that combines human hand motion retargeting with a task-oriented residual action policy, improving task performance by bridging the embodiment gap between human and robotic dexterous hands. Specifically, DexH2R learns the residual policy directly from retargeted primitive actions and task-oriented rewards, eliminating the need for labor-intensive teleoperation systems. Moreover, we incorporate test-time guidance for novel scenarios by taking in desired trajectories of human hands and objects, allowing the dexterous hand to acquire new skills with high generalizability. Extensive experiments in both simulation and real-world environments demonstrate the effectiveness of our work, outperforming prior state-of-the-arts by 40% across various settings.