ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection

TL;DR

ScrewMimic introduces a novel screw space action representation enabling robots to learn complex bimanual manipulation behaviors from human videos and refine them through self-supervised fine-tuning, outperforming baseline methods.

Contribution

The paper proposes a new screw space action model for bimanual manipulation, facilitating learning from human videos and self-supervised policy refinement.

Findings

Learns complex bimanual behaviors from a single demonstration.

Outperforms baseline methods in imitation and fine-tuning.

Effective in self-supervised policy refinement.

Abstract

Bimanual manipulation is a longstanding challenge in robotics due to the large number of degrees of freedom and the strict spatial and temporal synchronization required to generate meaningful behavior. Humans learn bimanual manipulation skills by watching other humans and by refining their abilities through play. In this work, we aim to enable robots to learn bimanual manipulation behaviors from human video demonstrations and fine-tune them through interaction. Inspired by seminal work in psychology and biomechanics, we propose modeling the interaction between two hands as a serial kinematic linkage -- as a screw motion, in particular, that we use to define a new action space for bimanual manipulation: screw actions. We introduce ScrewMimic, a framework that leverages this novel action representation to facilitate learning from human demonstration and self-supervised policy fine-tuning. Our experiments demonstrate that ScrewMimic is able to learn several complex bimanual behaviors from a single human video demonstration, and that it outperforms baselines that interpret demonstrations and fine-tune directly in the original space of motion of both arms. For more information and video results, https://robin-lab.cs.utexas.edu/ScrewMimic/