Causal Deep Q Network

TL;DR

This paper introduces a causal-enhanced Deep Q Network that incorporates causal reasoning to improve problem-solving by reducing spurious correlations, demonstrating superior performance on standard benchmarks.

Contribution

It presents a novel framework integrating causal principles into DQNs using the PEACE formula, advancing reinforcement learning with causal inference.

Findings

Outperforms standard DQNs on benchmark environments

Enhances problem-solving by mitigating spurious correlations

Demonstrates the effectiveness of causal reasoning in RL

Abstract

Deep Q Networks (DQN) have shown remarkable success in various reinforcement learning tasks. However, their reliance on associative learning often leads to the acquisition of spurious correlations, hindering their problem-solving capabilities. In this paper, we introduce a novel approach to integrate causal principles into DQNs, leveraging the PEACE (Probabilistic Easy vAriational Causal Effect) formula for estimating causal effects. By incorporating causal reasoning during training, our proposed framework enhances the DQN's understanding of the underlying causal structure of the environment, thereby mitigating the influence of confounding factors and spurious correlations. We demonstrate that integrating DQNs with causal capabilities significantly enhances their problem-solving capabilities without compromising performance. Experimental results on standard benchmark environments showcase that our approach outperforms conventional DQNs, highlighting the effectiveness of causal reasoning in reinforcement learning. Overall, our work presents a promising avenue for advancing the capabilities of deep reinforcement learning agents through principled causal inference.