Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

TL;DR

This paper introduces a method for explaining reinforcement learning agents by contrasting the expected consequences of different policies and translating states into understandable descriptions, enhancing transparency.

Contribution

It presents a novel approach to generate contrastive explanations for RL agents based on expected consequences and user-friendly state descriptions.

Findings

Users prefer explanations about policies over individual actions

The method effectively contrasts learned and queried policies

Pilot survey shows increased user understanding and trust

Abstract

Machine Learning models become increasingly proficient in complex tasks. However, even for experts in the field, it can be difficult to understand what the model learned. This hampers trust and acceptance, and it obstructs the possibility to correct the model. There is therefore a need for transparency of machine learning models. The development of transparent classification models has received much attention, but there are few developments for achieving transparent Reinforcement Learning (RL) models. In this study we propose a method that enables a RL agent to explain its behavior in terms of the expected consequences of state transitions and outcomes. First, we define a translation of states and actions to a description that is easier to understand for human users. Second, we developed a procedure that enables the agent to obtain the consequences of a single action, as well as its entire policy. The method calculates contrasts between the consequences of a policy derived from a user query, and of the learned policy of the agent. Third, a format for generating explanations was constructed. A pilot survey study was conducted to explore preferences of users for different explanation properties. Results indicate that human users tend to favor explanations about policy rather than about single actions.