Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning

TL;DR

This paper introduces PolicyExplainer, a visual analytics tool that helps humans understand reinforcement learning agents' decisions through visualizations, improving trust and comprehension especially for non-experts in safety-critical domains.

Contribution

The paper presents PolicyExplainer, a novel visual interface for RL explanations, validated through qualitative and quantitative assessments across multiple domains.

Findings

PolicyExplainer enhances understanding of RL decisions.

It promotes greater trust compared to text-based explanations.

Validated by domain practitioners for safety-critical applications.

Abstract

Reinforcement learning (RL) is used in many domains, including autonomous driving, robotics, stock trading, and video games. Unfortunately, the black box nature of RL agents, combined with legal and ethical considerations, makes it increasingly important that humans (including those are who not experts in RL) understand the reasoning behind the actions taken by an RL agent, particularly in safety-critical domains. To help address this challenge, we introduce PolicyExplainer, a visual analytics interface which lets the user directly query an autonomous agent. PolicyExplainer visualizes the states, policy, and expected future rewards for an agent, and supports asking and answering questions such as: Why take this action? Why not take this other action? When is this action taken? PolicyExplainer is designed based upon a domain analysis with RL researchers, and is evaluated via qualitative and quantitative assessments on a trio of domains: taxi navigation, a stack bot domain, and drug recommendation for HIV patients. We find that PolicyExplainer promotes trust and understanding of agent decisions better than a state-of-the-art text-based explanation approach. Interviews with domain practitioners provide further validation for PolicyExplainer as applied to safety-critical domains. Our results help demonstrate how visualization-based approaches can be leveraged to decode the behavior of autonomous RL agents, particularly for RL non-experts.