Experiential Explanations for Reinforcement Learning

TL;DR

This paper introduces Experiential Explanations, a novel method for making reinforcement learning agents more interpretable by generating counterfactual explanations that reveal how different rewards influence decisions, improving human understanding.

Contribution

The paper proposes a new technique using influence predictors to generate counterfactual explanations for RL agents, enhancing interpretability and user comprehension.

Findings

Participants better predicted agent actions with Experiential Explanations.

Participants found these explanations more understandable and useful.

Experiential Explanations improved perceived explanation quality.

Abstract

Reinforcement learning (RL) systems can be complex and non-interpretable, making it challenging for non-AI experts to understand or intervene in their decisions. This is due in part to the sequential nature of RL in which actions are chosen because of their likelihood of obtaining future rewards. However, RL agents discard the qualitative features of their training, making it difficult to recover user-understandable information for "why" an action is chosen. We propose a technique Experiential Explanations to generate counterfactual explanations by training influence predictors along with the RL policy. Influence predictors are models that learn how different sources of reward affect the agent in different states, thus restoring information about how the policy reflects the environment. Two human evaluation studies revealed that participants presented with Experiential Explanations were better able to correctly guess what an agent would do than those presented with other standard types of explanation. Participants also found that Experiential Explanations are more understandable, satisfying, complete, useful, and accurate. Qualitative analysis provides information on the factors of Experiential Explanations that are most useful and the desired characteristics that participants seek from the explanations.