DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C., Karen Liu

TL;DR
DexCap introduces a portable hand motion capture system and a novel imitation learning algorithm, DexIL, enabling robots to learn dexterous manipulation skills directly from human mocap data with high accuracy and robustness.
Contribution
The paper presents DexCap, a portable mocap system, and DexIL, a new imitation algorithm, advancing robot dexterity by effectively utilizing in-the-wild human hand motion data.
Findings
DexCap achieves occlusion-resistant, precise hand motion tracking.
DexIL successfully learns complex manipulation tasks from mocap data.
The system outperforms existing methods in six dexterous manipulation benchmarks.
Abstract
Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobot Manipulation and Learning · Teleoperation and Haptic Systems · Hand Gesture Recognition Systems
