Learning Reward Functions for Robotic Manipulation by Observing Humans

TL;DR

This paper introduces a method to learn general reward functions for robotic manipulation by observing unlabeled human videos, enabling robots to better explore and learn tasks without task-specific demonstrations.

Contribution

The work presents a novel approach to derive task-agnostic reward functions from human videos, generalizing across robot embodiments and environments without requiring task-specific data.

Findings

The learned reward function generalizes to unseen robot and environment configurations.

The method accelerates reinforcement learning for manipulation tasks in simulation.

It does not require task-specific human demonstrations or predefined correspondences.

Abstract

Observing a human demonstrator manipulate objects provides a rich, scalable and inexpensive source of data for learning robotic policies. However, transferring skills from human videos to a robotic manipulator poses several challenges, not least a difference in action and observation spaces. In this work, we use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. Thanks to the diversity of this training data, the learned reward function sufficiently generalizes to image observations from a previously unseen robot embodiment and environment to provide a meaningful prior for directed exploration in reinforcement learning. We propose two methods for scoring states relative to a goal image: through direct temporal regression, and through distances in an embedding space obtained with time-contrastive learning. By conditioning the function on a goal image, we are able to reuse one model across a variety of tasks. Unlike prior work on leveraging human videos to teach robots, our method, Human Offline Learned Distances (HOLD) requires neither a priori data from the robot environment, nor a set of task-specific human demonstrations, nor a predefined notion of correspondence across morphologies, yet it is able to accelerate training of several manipulation tasks on a simulated robot arm compared to using only a sparse reward obtained from task completion.