Hierarchical Reinforcement Learning with Targeted Causal Interventions

TL;DR

This paper introduces a causal graph-based approach to hierarchical reinforcement learning, enabling targeted interventions on subgoals, which significantly improves training efficiency and outperforms existing methods.

Contribution

It models subgoal structures as causal graphs and develops a causal discovery algorithm to enhance exploration and efficiency in HRL.

Findings

Significant reduction in training cost with targeted interventions.

Formal analysis of causal HRL for tree and Erdős-Rényi graph structures.

Outperforms existing methods on HRL tasks.

Abstract

Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erd\H{o}s-R\'enyi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.