On-Robot Reinforcement Learning with Goal-Contrastive Rewards

TL;DR

This paper introduces GCR, a novel reward learning method from passive videos that improves sample efficiency in robot reinforcement learning and enables cross-embodiment transfer, reducing the need for manual reward engineering.

Contribution

GCR is a new dense reward learning approach from passive videos that enhances RL efficiency and supports transfer across different embodiments.

Findings

GCR doubles the number of solvable tasks compared to baseline methods.

GCR enables effective cross-embodiment transfer from human and robot videos.

GCR improves sample efficiency in simulated and real robot environments.

Abstract

Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world. Unfortunately, RL can be prohibitively expensive, in terms of on-robot runtime, due to inefficient exploration when learning from a sparse reward signal. Designing dense reward functions is labour-intensive and requires domain expertise. In our work, we propose GCR (Goal-Contrastive Rewards), a dense reward function learning method that can be trained on passive video demonstrations. By using videos without actions, our method is easier to scale, as we can use arbitrary videos. GCR combines two loss functions, an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories. We perform experiments in simulated manipulation environments across RoboMimic and MimicGen tasks, as well as in the real world using a Franka arm and a Spot quadruped. We find that GCR leads to a more-sample efficient RL, enabling model-free RL to solve about twice as many tasks as our baseline reward learning methods. We also demonstrate positive cross-embodiment transfer from videos of people and of other robots performing a task. Website: https://gcr-robot.github.io/.