Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Models

TL;DR

This paper introduces Real2Gen, a method that uses a single human demonstration to generate unlimited training data in simulation, enabling effective robot manipulation policies with improved success rates and zero-shot real-world deployment.

Contribution

Real2Gen is a novel approach that transfers a single human demonstration into a simulation environment to generate extensive training data for imitation learning.

Findings

26.6% average increase in success rate

Better generalization of policies

Zero-shot real-world deployment achieved

Abstract

Imitation learning is a popular paradigm to teach robots new tasks, but collecting robot demonstrations through teleoperation or kinesthetic teaching is tedious and time-consuming. In contrast, directly demonstrating a task using our human embodiment is much easier and data is available in abundance, yet transfer to the robot can be non-trivial. In this work, we propose Real2Gen to train a manipulation policy from a single human demonstration. Real2Gen extracts required information from the demonstration and transfers it to a simulation environment, where a programmable expert agent can demonstrate the task arbitrarily many times, generating an unlimited amount of data to train a flow matching policy. We evaluate Real2Gen on human demonstrations from three different real-world tasks and compare it to a recent baseline. Real2Gen shows an average increase in the success rate of 26.6% and better generalization of the trained policy due to the abundance and diversity of training data. We further deploy our purely simulation-trained policy zero-shot in the real world. We make the data, code, and trained models publicly available at real2gen.cs.uni-freiburg.de.