Causal Explanation for Reinforcement Learning: Quantifying State and Temporal Importance

TL;DR

This paper introduces a causal explanation mechanism for reinforcement learning that quantifies the importance of states and their temporal influence on actions, providing more human-aligned explanations than traditional associational methods.

Contribution

It develops a novel causal explanation approach for RL policies and demonstrates its advantages over existing associational methods through multiple simulation studies.

Findings

Causal importance measures outperform associational methods in explaining RL policies.

The mechanism provides clearer, more human-aligned explanations of state-action relationships.

Simulation results include applications in crop irrigation, Blackjack, collision avoidance, and lunar lander.

Abstract

Explainability plays an increasingly important role in machine learning. Furthermore, humans view the world through a causal lens and thus prefer causal explanations over associational ones. Therefore, in this paper, we develop a causal explanation mechanism that quantifies the causal importance of states on actions and such importance over time. We also demonstrate the advantages of our mechanism over state-of-the-art associational methods in terms of RL policy explanation through a series of simulation studies, including crop irrigation, Blackjack, collision avoidance, and lunar lander.