R3M: A Universal Visual Representation for Robot Manipulation

TL;DR

This paper introduces R3M, a universal visual representation trained on human videos, that significantly enhances data-efficient robotic manipulation learning both in simulation and real-world tasks.

Contribution

The paper presents R3M, a novel pre-trained visual representation for robot manipulation, leveraging diverse human video data and multiple learning techniques, outperforming existing methods.

Findings

R3M improves task success by over 20% compared to training from scratch.

R3M outperforms state-of-the-art visual representations like CLIP and MoCo.

R3M enables real robot learning with just 20 demonstrations.

Abstract

We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks. Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations. The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning. Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and MoCo. Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations. Code and pre-trained models are available at https://tinyurl.com/robotr3m.