Let Me Show You: Learning by Retrieving from Egocentric Video for Robotic Manipulation

TL;DR

This paper introduces a novel robot learning method called Retrieving-from-Video (RfV), which leverages human demonstration videos and mid-level information to improve robotic manipulation and generalization in complex environments.

Contribution

The paper presents a new approach that uses a video retrieval system and mid-level features from human demonstrations to enhance robot policy learning and generalization.

Findings

Significant performance improvements over traditional methods.

Effective generalization to unseen tasks.

Successful deployment in real-world robotic scenarios.

Abstract

Robots operating in complex and uncertain environments face considerable challenges. Advanced robotic systems often rely on extensive datasets to learn manipulation tasks. In contrast, when humans are faced with unfamiliar tasks, such as assembling a chair, a common approach is to learn by watching video demonstrations. In this paper, we propose a novel method for learning robot policies by Retrieving-from-Video (RfV), using analogies from human demonstrations to address manipulation tasks. Our system constructs a video bank comprising recordings of humans performing diverse daily tasks. To enrich the knowledge from these videos, we extract mid-level information, such as object affordance masks and hand motion trajectories, which serve as additional inputs to enhance the robot model's learning and generalization capabilities. We further feature a dual-component system: a video retriever that taps into an external video bank to fetch task-relevant video based on task specification, and a policy generator that integrates this retrieved knowledge into the learning cycle. This approach enables robots to craft adaptive responses to various scenarios and generalize to tasks beyond those in the training data. Through rigorous testing in multiple simulated and real-world settings, our system demonstrates a marked improvement in performance over conventional robotic systems, showcasing a significant breakthrough in the field of robotics.