Hyperbolic Embeddings for Learning Options in Hierarchical Reinforcement Learning

TL;DR

This paper introduces a novel hyperbolic embedding approach for hierarchical reinforcement learning, enabling autonomous discovery of meaningful sub-tasks by exploiting the global topology of state spaces.

Contribution

It combines routing paradigms and graph-based skill discovery with hyperbolic embeddings to improve sub-task learning in hierarchical reinforcement learning.

Findings

Embeddings improve sub-task learning in discrete domains.

Embeddings enhance sub-task discovery in continuous domains.

Method enforces a global topology on states for better task decomposition.

Abstract

Hierarchical reinforcement learning deals with the problem of breaking down large tasks into meaningful sub-tasks. Autonomous discovery of these sub-tasks has remained a challenging problem. We propose a novel method of learning sub-tasks by combining paradigms of routing in computer networks and graph based skill discovery within the options framework to define meaningful sub-goals. We apply the recent advancements of learning embeddings using Riemannian optimisation in the hyperbolic space to embed the state set into the hyperbolic space and create a model of the environment. In doing so we enforce a global topology on the states and are able to exploit this topology to learn meaningful sub-tasks. We demonstrate empirically, both in discrete and continuous domains, how these embeddings can improve the learning of meaningful sub-tasks.