Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos

TL;DR

This paper introduces PSI, a framework that uses simulation-filtered human video data to train modular robot manipulation policies, improving task-specific grasping and manipulation without requiring robot data.

Contribution

The paper proposes a novel simulation-filtered learning approach for modular manipulation policies from human videos, enhancing grasp suitability and task performance.

Findings

Robust manipulation skills learned without robot data

Significant improvement over naive grasp generators

Effective transfer from human videos to robot manipulation

Abstract

The ability to learn manipulation skills by watching videos of humans has the potential to unlock a new source of highly scalable data for robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any robot data, resulting in significantly more robust performance than using a grasp generator naively.