Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning

TL;DR

This paper introduces VisionMask, a contrastive learning framework that generates explanations for reinforcement learning decisions by contrasting chosen actions with alternatives, improving interpretability and trust in RL agents.

Contribution

The paper presents a novel contrastive learning approach for explaining RL actions, addressing the gap in interpretability by explicitly contrasting actions in a self-supervised manner.

Findings

VisionMask improves human understanding of RL decisions

The method maintains high fidelity and robustness

It enables effective counterfactual analysis

Abstract

Reinforcement learning (RL) has demonstrated remarkable success in solving complex decision-making problems, yet its adoption in critical domains is hindered by the lack of interpretability in its decision-making processes. Existing explainable AI (xAI) approaches often fail to provide meaningful explanations for RL agents, particularly because they overlook the contrastive nature of human reasoning--answering "why this action instead of that one?". To address this gap, we propose a novel framework of contrastive learning to explain RL selected actions, named $\textbf{VisionMask}$. VisionMask is trained to generate explanations by explicitly contrasting the agent's chosen action with alternative actions in a given state using a self-supervised manner. We demonstrate the efficacy of our method through experiments across diverse RL environments, evaluating it in terms of faithfulness, robustness, and complexity. Our results show that VisionMask significantly improves human understanding of agent behavior while maintaining accuracy and fidelity. Furthermore, we present examples illustrating how VisionMask can be used for counterfactual analysis. This work bridges the gap between RL and xAI, paving the way for safer and more interpretable RL systems.