Hierarchical Skills for Efficient Exploration

TL;DR

This paper introduces a hierarchical skill learning framework for reinforcement learning that automatically balances general and specific skills, improving exploration and performance on diverse bipedal robot tasks.

Contribution

It proposes an unsupervised hierarchical skill learning method with a three-layered algorithm to adaptively trade off skill complexity for downstream tasks.

Findings

Outperforms current state-of-the-art hierarchical RL methods

Effectively balances general and specific skills in diverse tasks

Demonstrates improved exploration and learning efficiency

Abstract

In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for end- to-end hierarchical reinforcement learning and unsupervised skill discovery. Code and videos are available at https://facebookresearch.github.io/hsd3 .