Hierarchical Behaviour Spaces

TL;DR

Hierarchical Behaviour Spaces (HBS) leverage linear combinations of reward functions to create a rich behaviour space, enhancing exploration and performance in complex environments like NetHack.

Contribution

HBS introduces a novel way to use reward functions for inducing a diverse behaviour space, improving exploration over traditional hierarchical reinforcement learning methods.

Findings

HBS achieves strong performance on NetHack.

Benefits of hierarchy stem from increased exploration, not just long-term reasoning.

Linear reward combinations enable more expressive policies.

Abstract

Recent work in hierarchical reinforcement learning has shown success in scaling to billions of timesteps when learning over a set of predefined option reward functions. We show that, instead of using a single reward function per option, the reward functions can be effectively used to induce a space of behaviours, by letting the controller specify linear combinations over reward functions, allowing a more expressive set of policies to be represented. We call this method Hierarchical Behaviour Spaces (HBS). We evaluate HBS on the NetHack Learning Environment, demonstrating strong performance. We conduct a series of experiments and determine that, perhaps going against conventional wisdom, the benefits of hierarchy in our method come from increased exploration rather than long term reasoning.