D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection

TL;DR

D3HRL introduces a distributed hierarchical reinforcement learning method that leverages causal discovery and spurious correlation detection to improve decision-making in complex, long-horizon tasks.

Contribution

It presents a novel causal HRL framework that models delay effects, eliminates spurious correlations, and iteratively learns true causal relationships for hierarchical policy construction.

Findings

Outperforms existing HRL algorithms in sensitivity to delay effects

Accurately identifies causal relationships in complex environments

Demonstrates improved decision-making in 2D-MineCraft and MiniGrid

Abstract

Current Hierarchical Reinforcement Learning (HRL) algorithms excel in long-horizon sequential decision-making tasks but still face two challenges: delay effects and spurious correlations. To address them, we propose a causal HRL approach called D3HRL. First, D3HRL models delayed effects as causal relationships across different time spans and employs distributed causal discovery to learn these relationships. Second, it employs conditional independence testing to eliminate spurious correlations. Finally, D3HRL constructs and trains hierarchical policies based on the identified true causal relationships. These three steps are iteratively executed, gradually exploring the complete causal chain of the task. Experiments conducted in 2D-MineCraft and MiniGrid show that D3HRL demonstrates superior sensitivity to delay effects and accurately identifies causal relationships, leading to reliable decision-making in complex environments.