Self-Supervised Reinforcement Learning that Transfers using Random Features

TL;DR

This paper introduces a self-supervised reinforcement learning approach that leverages random features for reward modeling, enabling transfer across tasks without explicit reward labels and facilitating rapid adaptation in complex environments.

Contribution

It proposes a novel self-supervised pre-training method for model-free RL using random features, allowing implicit environment modeling and efficient transfer to new tasks.

Findings

Enables transfer across manipulation and locomotion tasks in simulation.

Allows fast adaptation to new reward functions without additional training.

Operates effectively with offline datasets and no reward labels.

Abstract

Model-free reinforcement learning algorithms have exhibited great potential in solving single-task sequential decision-making problems with high-dimensional observations and long horizons, but are known to be hard to generalize across tasks. Model-based RL, on the other hand, learns task-agnostic models of the world that naturally enables transfer across different reward functions, but struggles to scale to complex environments due to the compounding error. To get the best of both worlds, we propose a self-supervised reinforcement learning method that enables the transfer of behaviors across tasks with different rewards, while circumventing the challenges of model-based RL. In particular, we show self-supervised pre-training of model-free reinforcement learning with a number of random features as rewards allows implicit modeling of long-horizon environment dynamics. Then, planning techniques like model-predictive control using these implicit models enable fast adaptation to problems with new reward functions. Our method is self-supervised in that it can be trained on offline datasets without reward labels, but can then be quickly deployed on new tasks. We validate that our proposed method enables transfer across tasks on a variety of manipulation and locomotion domains in simulation, opening the door to generalist decision-making agents.