Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations

TL;DR

This paper introduces G4RL, a graph encoder-decoder that improves goal-conditioned hierarchical reinforcement learning by effectively utilizing graph representations of task structure, leading to better performance in various environments.

Contribution

It develops a graph encoder-decoder for unseen states, enhancing GCHRL by leveraging graph information without domain-specific knowledge, applicable in symmetric and reversible transition environments.

Findings

Significant performance improvements in dense and sparse reward settings.

Effective use of intrinsic rewards from the graph encoder-decoder.

Small additional computational cost for enhanced learning.

Abstract

The integration of graphs with Goal-conditioned Hierarchical Reinforcement Learning (GCHRL) has recently gained attention, as intermediate goals (subgoals) can be effectively sampled from graphs that naturally represent the overall task structure in most RL tasks. However, existing approaches typically rely on domain-specific knowledge to construct these graphs, limiting their applicability to new tasks. Other graph-based approaches create graphs dynamically during exploration but struggle to fully utilize them, because they have problems passing the information in the graphs to newly visited states. Additionally, current GCHRL methods face challenges such as sample inefficiency and poor subgoal representation. This paper proposes a solution to these issues by developing a graph encoder-decoder to evaluate unseen states. Our proposed method, Graph-Guided sub-Goal representation Generation RL (G4RL), can be incorporated into any existing GCHRL method when operating in environments with primarily symmetric and reversible transitions to enhance performance across this class of problems. We show that the graph encoder-decoder can be effectively implemented using a network trained on the state graph generated during exploration. Empirical results indicate that leveraging high and low-level intrinsic rewards from the graph encoder-decoder significantly enhances the performance of state-of-the-art GCHRL approaches with an extra small computational cost in dense and sparse reward environments.