SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policies

TL;DR

SAFE-RL introduces a saliency-aware counterfactual explanation framework for deep reinforcement learning policies, improving interpretability in safety-critical and high-dimensional environments by generating plausible, minimal, and informative counterfactuals.

Contribution

This work presents a novel saliency-guided method for generating counterfactual explanations in DRL, addressing challenges of high-dimensional inputs and temporal dependencies.

Findings

Outperforms state-of-the-art CF methods in diverse environments

Produces more plausible and informative counterfactuals

Effective across multiple DRL agents and tasks

Abstract

While Deep Reinforcement Learning (DRL) has emerged as a promising solution for intricate control tasks, the lack of explainability of the learned policies impedes its uptake in safety-critical applications, such as automated driving systems (ADS). Counterfactual (CF) explanations have recently gained prominence for their ability to interpret black-box Deep Learning (DL) models. CF examples are associated with minimal changes in the input, resulting in a complementary output by the DL model. Finding such alternations, particularly for high-dimensional visual inputs, poses significant challenges. Besides, the temporal dependency introduced by the reliance of the DRL agent action on a history of past state observations further complicates the generation of CF examples. To address these challenges, we propose using a saliency map to identify the most influential input pixels across the sequence of past observed states by the agent. Then, we feed this map to a deep generative model, enabling the generation of plausible CFs with constrained modifications centred on the salient regions. We evaluate the effectiveness of our framework in diverse domains, including ADS, Atari Pong, Pacman and space-invaders games, using traditional performance metrics such as validity, proximity and sparsity. Experimental results demonstrate that this framework generates more informative and plausible CFs than the state-of-the-art for a wide range of environments and DRL agents. In order to foster research in this area, we have made our datasets and codes publicly available at https://github.com/Amir-Samadi/SAFE-RL.