HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning

TL;DR

HAC Explore (HACx) combines hierarchical reinforcement learning with exploration bonuses to effectively solve complex sparse reward tasks, outperforming previous methods and achieving success in long-horizon continuous control challenges.

Contribution

This paper introduces HACx, the first RL method integrating hierarchical learning with exploration bonuses to solve long-horizon sparse reward tasks.

Findings

HACx outperforms individual components and previous methods.

HACx solves a sparse reward, continuous-control task with over 1,000 actions.

HACx is the first RL approach to tackle such complex long-horizon tasks.

Abstract

Sparse rewards and long time horizons remain challenging for reinforcement learning algorithms. Exploration bonuses can help in sparse reward settings by encouraging agents to explore the state space, while hierarchical approaches can assist with long-horizon tasks by decomposing lengthy tasks into shorter subtasks. We propose HAC Explore (HACx), a new method that combines these approaches by integrating the exploration bonus method Random Network Distillation (RND) into the hierarchical approach Hierarchical Actor-Critic (HAC). HACx outperforms either component method on its own, as well as an existing approach to combining hierarchy and exploration, in a set of difficult simulated robotics tasks. HACx is the first RL method to solve a sparse reward, continuous-control task that requires over 1,000 actions.