Masquerade: Learning from In-the-wild Human Videos using Data-Editing

TL;DR

Masquerade is a novel method that leverages in-the-wild human videos, through editing techniques, to significantly enhance robot policy learning and generalization in complex tasks.

Contribution

The paper introduces a data-editing pipeline that transforms human videos into robot demonstrations, enabling effective pre-training and fine-tuning of robot policies with limited real robot data.

Findings

Policies trained with Masquerade outperform baselines by 5-6x on unseen scenes.

Pre-training on 675K edited frames improves generalization.

Both robot overlay and co-training are essential for performance gains.

Abstract

Robot manipulation research still suffers from significant data scarcity: even the largest robot datasets are orders of magnitude smaller and less diverse than those that fueled recent breakthroughs in language and vision. We introduce Masquerade, a method that edits in-the-wild egocentric human videos to bridge the visual embodiment gap between humans and robots and then learns a robot policy with these edited videos. Our pipeline turns each human video into robotized demonstrations by (i) estimating 3-D hand poses, (ii) inpainting the human arms, and (iii) overlaying a rendered bimanual robot that tracks the recovered end-effector trajectories. Pre-training a visual encoder to predict future 2-D robot keypoints on 675K frames of these edited clips, and continuing that auxiliary loss while fine-tuning a diffusion policy head on only 50 robot demonstrations per task, yields policies that generalize significantly better than prior work. On three long-horizon, bimanual kitchen tasks evaluated in three unseen scenes each, Masquerade outperforms baselines by 5-6x. Ablations show that both the robot overlay and co-training are indispensable, and performance scales logarithmically with the amount of edited human video. These results demonstrate that explicitly closing the visual embodiment gap unlocks a vast, readily available source of data from human videos that can be used to improve robot policies.