Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning

TL;DR

This paper introduces a counterfactual reasoning approach to evaluate the explanations provided by saliency maps in deep reinforcement learning, highlighting their exploratory nature rather than definitive explanations.

Contribution

It proposes an empirical, counterfactual framework to test saliency map hypotheses in deep RL, providing a more rigorous evaluation method.

Findings

Saliency maps often lack falsifiability in explaining RL behavior.

Counterfactual analysis reveals limitations of saliency maps as explanations.

Saliency maps are more suitable for exploration than definitive explanation.

Abstract

Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.