DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos

TL;DR

DexImit is a framework that converts monocular human videos into robot manipulation data, enabling scalable learning of bimanual dexterous tasks without additional information.

Contribution

It introduces a four-stage pipeline to generate physically plausible robot data from human videos, bridging the embodiment gap for improved robot manipulation learning.

Findings

Capable of handling diverse manipulation tasks

Enables zero-shot real-world deployment

Generates large-scale robot data from human videos

Abstract

Data scarcity fundamentally limits the generalization of bimanual dexterous manipulation, as real-world data collection for dexterous hands is expensive and labor-intensive. Human manipulation videos, as a direct carrier of manipulation knowledge, offer significant potential for scaling up robot learning. However, the substantial embodiment gap between human hands and robotic dexterous hands makes direct pretraining from human videos extremely challenging. To bridge this gap and unleash the potential of large-scale human manipulation video data, we propose DexImit, an automated framework that converts monocular human manipulation videos into physically plausible robot data, without any additional information. DexImit employs a four-stage generation pipeline: (1) reconstructing hand-object interactions from arbitrary viewpoints with near-metric scale; (2) performing subtask decomposition and bimanual scheduling; (3) synthesizing robot trajectories consistent with the demonstrated interactions; (4) comprehensive data augmentation for zero-shot real-world deployment. Building on these designs, DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models. DexImit is capable of handling diverse manipulation tasks, including tool use (e.g., cutting an apple), long-horizon tasks (e.g., making a beverage), and fine-grained manipulations (e.g., stacking cups).