Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

TL;DR

This paper introduces a hierarchical reinforcement learning method that autonomously discovers symbolic goal representations through reachability analysis, improving data efficiency and interpretability in navigation tasks.

Contribution

It proposes a novel developmental mechanism for subgoal discovery that learns environment state abstractions without manual goal specification.

Findings

Learned representations are interpretable.

Achieves improved data efficiency.

Effective in navigation tasks.

Abstract

Open-ended learning benefits immensely from the use of symbolic methods for goal representation as they offer ways to structure knowledge for efficient and transferable learning. However, the existing Hierarchical Reinforcement Learning (HRL) approaches relying on symbolic reasoning are often limited as they require a manual goal representation. The challenge in autonomously discovering a symbolic goal representation is that it must preserve critical information, such as the environment dynamics. In this work, we propose a developmental mechanism for subgoal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states that have similar roles in the task. We create a HRL algorithm that gradually learns this representation along with the policies and evaluate it on navigation tasks to show the learned representation is interpretable and results in data efficiency.