EgoMimic: Scaling Imitation Learning via Egocentric Video

TL;DR

EgoMimic introduces a comprehensive framework that leverages egocentric human videos and 3D hand tracking to significantly improve manipulation tasks in robots, enabling better generalization and scaling of imitation learning.

Contribution

The paper presents a full-stack system combining data collection, alignment, and co-training that treats human and robot data equally for imitation learning, advancing beyond high-level intent extraction.

Findings

EgoMimic outperforms prior imitation methods on various manipulation tasks.

Adding 1 hour of human hand data yields more improvement than 1 hour of robot data.

The approach enables generalization to new scenes and tasks.

Abstract

The scale and diversity of demonstration data required for imitation learning is a significant challenge. We present EgoMimic, a full-stack framework which scales manipulation via human embodiment data, specifically egocentric human videos paired with 3D hand tracking. EgoMimic achieves this through: (1) a system to capture human embodiment data using the ergonomic Project Aria glasses, (2) a low-cost bimanual manipulator that minimizes the kinematic gap to human data, (3) cross-domain data alignment techniques, and (4) an imitation learning architecture that co-trains on human and robot data. Compared to prior works that only extract high-level intent from human videos, our approach treats human and robot data equally as embodied demonstration data and learns a unified policy from both data sources. EgoMimic achieves significant improvement on a diverse set of long-horizon, single-arm and bimanual manipulation tasks over state-of-the-art imitation learning methods and enables generalization to entirely new scenes. Finally, we show a favorable scaling trend for EgoMimic, where adding 1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data. Videos and additional information can be found at https://egomimic.github.io/