Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning

TL;DR

This paper introduces a multi-goal hierarchical reinforcement learning algorithm that uses a high-level manager to set multiple subgoals for the low-level worker, significantly improving exploration efficiency in sparse reward environments like Montezuma's Revenge.

Contribution

The paper proposes a novel multi-goal HRL framework with a high-level policy providing multiple subgoals, enabling faster exploration without manual environment-specific design.

Findings

Achieves state-of-the-art performance in Montezuma's Revenge

Reduces training time compared to existing HRL methods

Effectively learns to respond to multiple subgoals from high-level policies

Abstract

Hierarchical Reinforcement Learning (HRL) exploits temporally extended actions, or options, to make decisions from a higher-dimensional perspective to alleviate the sparse reward problem, one of the most challenging problems in reinforcement learning. The majority of existing HRL algorithms require either significant manual design with respect to the specific environment or enormous exploration to automatically learn options from data. To achieve fast exploration without using manual design, we devise a multi-goal HRL algorithm, consisting of a high-level policy Manager and a low-level policy Worker. The Manager provides the Worker multiple subgoals at each time step. Each subgoal corresponds to an option to control the environment. Although the agent may show some confusion at the beginning of training since it is guided by three diverse subgoals, the agent's behavior policy will quickly learn how to respond to multiple subgoals from the high-level controller on different occasions. By exploiting multiple subgoals, the exploration efficiency is significantly improved. We conduct experiments in Atari's Montezuma's Revenge environment, a well-known sparse reward environment, and in doing so achieve the same performance as state-of-the-art HRL methods with substantially reduced training time cost.