Accelerating Goal-Conditioned RL Algorithms and Research

TL;DR

This paper introduces JaxGCRL, a high-performance benchmark and codebase for self-supervised goal-conditioned reinforcement learning, significantly accelerating training and facilitating research in the field.

Contribution

It provides a GPU-accelerated, stable RL algorithm and benchmark for self-supervised GCRL, enabling rapid training and evaluation of agents in complex environments.

Findings

Training time reduced by up to 22 times

Identified key design choices that stabilize training

Enabled training for millions of environment steps in minutes

Abstract

Self-supervision has the potential to transform reinforcement learning (RL), paralleling the breakthroughs it has enabled in other areas of machine learning. While self-supervised learning in other domains aims to find patterns in a fixed dataset, self-supervised goal-conditioned reinforcement learning (GCRL) agents discover new behaviors by learning from the goals achieved during unstructured interaction with the environment. However, these methods have failed to see similar success, both due to a lack of data from slow environment simulations as well as a lack of stable algorithms. We take a step toward addressing both of these issues by releasing a high-performance codebase and benchmark (JaxGCRL) for self-supervised GCRL, enabling researchers to train agents for millions of environment steps in minutes on a single GPU. By utilizing GPU-accelerated replay buffers, environments, and a stable contrastive RL algorithm, we reduce training time by up to $22\times$. Additionally, we assess key design choices in contrastive RL, identifying those that most effectively stabilize and enhance training performance. With this approach, we provide a foundation for future research in self-supervised GCRL, enabling researchers to quickly iterate on new ideas and evaluate them in diverse and challenging environments. Website + Code: https://github.com/MichalBortkiewicz/JaxGCRL