Delay-Empowered Causal Hierarchical Reinforcement Learning

TL;DR

DECHRL is a novel hierarchical reinforcement learning method that explicitly models causal structures and stochastic delays to improve decision-making in environments with temporal uncertainty.

Contribution

It introduces a delay-aware empowerment objective within hierarchical RL that explicitly incorporates causal and delay modeling for better handling of temporal delays.

Findings

DECHRL effectively models stochastic delays in environments.

It significantly outperforms baseline methods in delay-affected tasks.

Experimental results demonstrate improved decision-making under temporal uncertainty.

Abstract

Many real-world tasks involve delayed effects, where the outcomes of actions emerge after varying time lags. Existing delay-aware reinforcement learning methods often rely on state augmentation, prior knowledge of delay distributions, or access to non-delayed data, limiting their generalization. Hierarchical reinforcement learning, by contrast, inherently offers advantages in handling delays due to its hierarchical structure, yet existing methods are restricted to fixed delays. To address these limitations, we propose Delay-Empowered Causal Hierarchical Reinforcement Learning (DECHRL). DECHRL explicitly models both the causal structure of state transitions and their associated stochastic delay distributions. These are then incorporated into a delay-aware empowerment objective that drives proactive exploration toward highly controllable states, thereby improving performance under temporal uncertainty. We evaluate DECHRL in modified 2D-Minecraft and MiniGrid environments featuring stochastic delays. Experimental results show that DECHRL effectively models temporal delays and significantly outperforms baselines in decision-making under temporal uncertainty.