A Survey on Causal Reinforcement Learning

TL;DR

This survey reviews recent advances in Causal Reinforcement Learning (CRL), highlighting how causality concepts are integrated to improve RL's data efficiency and interpretability across various models and applications.

Contribution

It systematically categorizes CRL approaches based on prior causality information and analyzes their formal models, evaluation methods, and future research directions.

Findings

CRL approaches are divided into two main categories based on causality information availability.

The survey covers formal models including MDP, POMDP, MAB, and DTR.

Emerging applications and open-source tools are summarized for future exploration.

Abstract

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR). Moreover, we summarize the evaluation matrices and open sources while we discuss emerging applications, along with promising prospects for the future development of CRL.