DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

TL;DR

DexCap introduces a portable hand motion capture system and a novel imitation learning algorithm, DexIL, enabling robots to learn dexterous manipulation skills directly from human mocap data with high accuracy and robustness.

Contribution

The paper presents DexCap, a portable mocap system, and DexIL, a new imitation algorithm, advancing robot dexterity by effectively utilizing in-the-wild human hand motion data.

Findings

DexCap achieves occlusion-resistant, precise hand motion tracking.

DexIL successfully learns complex manipulation tasks from mocap data.

The system outperforms existing methods in six dexterous manipulation benchmarks.

Abstract

Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io