Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

TL;DR

This paper introduces a framework for learning representations in hierarchical reinforcement learning that optimizes the expected reward of the best hierarchical policy, leading to improved performance in complex continuous-control tasks.

Contribution

The paper develops a novel notion of sub-optimality for representations and derives bounds that inform practical learning objectives for hierarchical RL.

Findings

Better representations lead to improved hierarchical policies.

The approach outperforms existing methods on continuous-control tasks.

Qualitative improvements in learned representations are observed.

Abstract

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at https://sites.google.com/view/representation-hrl).