SHIRO: Soft Hierarchical Reinforcement Learning

TL;DR

SHIRO introduces an entropy-maximizing hierarchical reinforcement learning algorithm that enhances exploration and learning efficiency in robotic control tasks by adding entropy at both hierarchy levels, supported by theoretical and empirical evidence.

Contribution

This work is the first to theoretically motivate and empirically validate the addition of entropy at both levels of hierarchical reinforcement learning.

Findings

Adding entropy to high-level policies improves exploration.

SHIRO outperforms state-of-the-art methods on robotic benchmarks.

High temperature in low-level policies increases stochasticity and learning difficulty.

Abstract

Hierarchical Reinforcement Learning (HRL) algorithms have been demonstrated to perform well on high-dimensional decision making and robotic control tasks. However, because they solely optimize for rewards, the agent tends to search the same space redundantly. This problem reduces the speed of learning and achieved reward. In this work, we present an Off-Policy HRL algorithm that maximizes entropy for efficient exploration. The algorithm learns a temporally abstracted low-level policy and is able to explore broadly through the addition of entropy to the high-level. The novelty of this work is the theoretical motivation of adding entropy to the RL objective in the HRL setting. We empirically show that the entropy can be added to both levels if the Kullback-Leibler (KL) divergence between consecutive updates of the low-level policy is sufficiently small. We performed an ablative study to analyze the effects of entropy on hierarchy, in which adding entropy to high-level emerged as the most desirable configuration. Furthermore, a higher temperature in the low-level leads to Q-value overestimation and increases the stochasticity of the environment that the high-level operates on, making learning more challenging. Our method, SHIRO, surpasses state-of-the-art performance on a range of simulated robotic control benchmark tasks and requires minimal tuning.