Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling

TL;DR

This paper introduces a delay-aware reinforcement learning approach that models implicit causal graphs to improve transferability and adaptation across tasks with delayed feedback.

Contribution

It proposes a novel method using implicit causal graph modeling and structured representations to enable effective cross-task transfer and rapid policy adaptation in delayed environments.

Findings

Outperforms baseline methods on DMC tasks with random delays.

Effectively transfers structured representations to new tasks.

Accelerates policy adaptation in cross-task scenarios.

Abstract

Random delays weaken the temporal correspondence between actions and subsequent state feedback, making it difficult for agents to identify the true propagation process of action effects. In cross-task scenarios, changes in task objectives and reward formulations further reduce the reusability of previously acquired task knowledge. To address this problem, this paper proposes a transferable delay-aware reinforcement learning method based on implicit causal graph modeling. The proposed method uses a field-node encoder to represent high-dimensional observations as latent states with node-level semantics, and employs a message-passing mechanism to characterize dynamic causal dependencies among nodes, thereby learning transferable structured representations and environment dynamics knowledge. On this basis, imagination-driven behavior learning and planning are incorporated to optimize policies in the latent space, enabling cross-task knowledge transfer and rapid adaptation. Experimental results show that the proposed method outperforms baseline methods on DMC continuous control tasks with random delays. Cross-task transfer experiments further demonstrate that the learned structured representations and dynamics knowledge can be effectively transferred to new tasks and significantly accelerate policy adaptation.