Vision-based Manipulation from Single Human Video with Open-World Object Graphs

TL;DR

This paper introduces ORION, an object-centric algorithm enabling robots to learn manipulation skills from a single human video demonstration, generalizing across diverse environments and objects.

Contribution

The work presents a novel approach that extracts manipulation plans from single videos and conditions policies on these plans, advancing open-world robot imitation learning.

Findings

Achieved an average success rate of 74.4% across tasks.

Effective in generalizing to new objects and environments.

Works with RGB and RGB-D videos for manipulation learning.

Abstract

This work presents an object-centric approach to learning vision-based manipulation skills from human videos. We investigate the problem of robot manipulation via imitation in the open-world setting, where a robot learns to manipulate novel objects from a single video demonstration. We introduce ORION, an algorithm that tackles the problem by extracting an object-centric manipulation plan from a single RGB or RGB-D video and deriving a policy that conditions on the extracted plan. Our method enables the robot to learn from videos captured by daily mobile devices and to generalize the policies to deployment environments with varying visual backgrounds, camera angles, spatial layouts, and novel object instances. We systematically evaluate our method on both short-horizon and long-horizon tasks, using RGB-D and RGB-only demonstration videos. Across varied tasks and demonstration types (RGB-D / RGB), we observe an average success rate of 74.4%, demonstrating the efficacy of ORION in learning from a single human video in the open world. Additional materials can be found on our project website: https://ut-austin-rpl.github.io/ORION-release.