Learning to Transfer Human Hand Skills for Robot Manipulations

TL;DR

This paper introduces a novel method for transferring human hand manipulation skills to robots by learning a joint motion manifold and generating pseudo-supervision triplets, resulting in improved robot dexterity in manipulation tasks.

Contribution

It proposes a new data-driven approach that infers plausible robot actions from human demonstrations, addressing the embodiment gap between humans and robots.

Findings

Significantly outperforms conventional retargeting methods.

Effectively bridges the embodiment gap in robot manipulation.

Demonstrated successful real-world robot hand manipulation.

Abstract

We present a method for teaching dexterous manipulation tasks to robots from human hand motion demonstrations. Unlike existing approaches that solely rely on kinematics information without taking into account the plausibility of robot and object interaction, our method directly infers plausible robot manipulation actions from human motion demonstrations. To address the embodiment gap between the human hand and the robot system, our approach learns a joint motion manifold that maps human hand movements, robot hand actions, and object movements in 3D, enabling us to infer one motion component from others. Our key idea is the generation of pseudo-supervision triplets, which pair human, object, and robot motion trajectories synthetically. Through real-world experiments with robot hand manipulation, we demonstrate that our data-driven retargeting method significantly outperforms conventional retargeting techniques, effectively bridging the embodiment gap between human and robotic hands. Website at https://rureadyo.github.io/MocapRobot/.