Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations

TL;DR

This paper presents RIGVid, a system enabling robots to learn complex manipulation tasks by imitating AI-generated videos filtered by vision-language models, eliminating the need for physical demonstrations.

Contribution

The work introduces a novel approach where AI-generated videos serve as supervision for robotic manipulation, bypassing traditional physical training or demonstrations.

Findings

Filtered generated videos are as effective as real demonstrations.

Performance improves with higher quality video generation.

Generated videos outperform keypoint prediction methods.

Abstract

This work introduces Robots Imitating Generated Videos (RIGVid), a system that enables robots to perform complex manipulation tasks--such as pouring, wiping, and mixing--purely by imitating AI-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out results that do not follow the command. A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion. Through extensive real-world evaluations, we show that filtered generated videos are as effective as real demonstrations, and that performance improves with generation quality. We also show that relying on generated videos outperforms more compact alternatives such as keypoint prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These findings suggest that videos produced by a state-of-the-art off-the-shelf model can offer an effective source of supervision for robotic manipulation.