Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective

TL;DR

This paper introduces A2CR, an interpretable reinforcement learning framework that enhances transparency by classifying and explaining agent actions, demonstrated through improved interpretability and purpose-based saliency in game environments.

Contribution

The paper presents a novel A2CR model integrating a Reasoner network into Actor-Critic RL, enabling automatic interpretation of agent behavior and purpose classification.

Findings

Reasoner label proportions vary with exploration levels in games

Purpose-based saliencies are more focused and understandable

A2CR improves interpretability of RL agents in complex environments

Abstract

Reinforcement learning (RL) is a powerful tool for solving complex decision-making problems, but its lack of transparency and interpretability has been a major challenge in domains where decisions have significant real-world consequences. In this paper, we propose a novel Advantage Actor-Critic with Reasoner (A2CR), which can be easily applied to Actor-Critic-based RL models and make them interpretable. A2CR consists of three interconnected networks: the Policy Network, the Value Network, and the Reasoner Network. By predefining and classifying the underlying purpose of the actor's actions, A2CR automatically generates a more comprehensive and interpretable paradigm for understanding the agent's decision-making process. It offers a range of functionalities such as purpose-based saliency, early failure detection, and model supervision, thereby promoting responsible and trustworthy RL. Evaluations conducted in action-rich Super Mario Bros environments yield intriguing findings: Reasoner-predicted label proportions decrease for ``Breakout" and increase for ``Hovering" as the exploration level of the RL algorithm intensifies. Additionally, purpose-based saliencies are more focused and comprehensible.