Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration

TL;DR

This paper introduces SUPE, a method that leverages unlabeled offline data to learn exploration strategies in reinforcement learning by extracting skills, pseudo-labeling data, and combining these for efficient online exploration, outperforming prior methods.

Contribution

The paper proposes a novel approach that combines skill extraction and pseudo-labeling of unlabeled data to improve exploration in RL, demonstrating significant performance gains.

Findings

SUPE outperforms prior strategies on 42 long-horizon tasks.

Combining skill learning with pseudo-labeled data enhances exploration.

Transforming prior data into high-level, task-relevant examples is effective.

Abstract

Unsupervised pretraining has been transformative in many supervised domains. However, applying such ideas to reinforcement learning (RL) presents a unique challenge in that fine-tuning does not involve mimicking task-specific data, but rather exploring and locating the solution through iterative self-improvement. In this work, we study how unlabeled offline trajectory data can be leveraged to learn efficient exploration strategies. While prior data can be used to pretrain a set of low-level skills, or as additional off-policy data for online RL, it has been unclear how to combine these ideas effectively for online exploration. Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combination of these ideas compounds their benefits. Our method first extracts low-level skills using a variational autoencoder (VAE), and then pseudo-labels unlabeled trajectories with optimistic rewards and high-level action labels, transforming prior data into high-level, task-relevant examples that encourage novelty-seeking behavior. Finally, SUPE uses these transformed examples as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently. In our experiments, SUPE consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.