Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation

TL;DR

This paper introduces a counterfactual explanation framework for deep reinforcement learning models, enhancing interpretability and verification in safety-critical applications like autonomous driving and gaming.

Contribution

It presents a novel counterfactual explanation method using policy distillation to interpret black-box DRL decisions, validated through experiments in driving and gaming domains.

Findings

Framework produces plausible explanations for DRL decisions

Effective in safety-critical system verification

Applicable to diverse DRL applications

Abstract

Deep Reinforcement Learning (DRL) has demonstrated promising capability in solving complex control problems. However, DRL applications in safety-critical systems are hindered by the inherent lack of robust verification techniques to assure their performance in such applications. One of the key requirements of the verification process is the development of effective techniques to explain the system functionality, i.e., why the system produces specific results in given circumstances. Recently, interpretation methods based on the Counterfactual (CF) explanation approach have been proposed to address the problem of explanation in DRLs. This paper proposes a novel CF explanation framework to explain the decisions made by a black-box DRL. To evaluate the efficacy of the proposed explanation framework, we carried out several experiments in the domains of automated driving systems and Atari Pong game. Our analysis demonstrates that the proposed framework generates plausible and meaningful explanations for various decisions made by deep underlying DRLs. Source codes are available at: \url{https://github.com/Amir-Samadi/Counterfactual-Explanation}