A Theoretical Framework for Explaining Reinforcement Learning with Shapley Values

TL;DR

This paper introduces SVERL, a theoretical framework using Shapley values to explain reinforcement learning agents' behaviour, outcomes, and predictions with mathematically justified, interpretable explanations.

Contribution

It develops a unified, axiomatic approach for explaining RL agents through feature influence, addressing interpretability and conceptual clarity issues.

Findings

SVERL provides precise, interpretable explanations of RL agents.

The framework identifies and corrects conceptual issues in prior explanations.

Illustrative examples demonstrate the usefulness of SVERL in understanding agent behaviour.

Abstract

Reinforcement learning agents can achieve super-human performance in complex decision-making tasks, but their behaviour is often difficult to understand and explain. This lack of explanation limits deployment, especially in safety-critical settings where understanding and trust are essential. We identify three core explanatory targets that together provide a comprehensive view of reinforcement learning agents: behaviour, outcomes, and predictions. We develop a unified theoretical framework for explaining these three elements of reinforcement learning agents through the influence of individual features that the agent observes in its environment. We derive feature influences by using Shapley values, which collectively and uniquely satisfy a set of well-motivated axioms for fair and consistent credit assignment. The proposed approach, Shapley Values for Explaining Reinforcement Learning (SVERL), provides a single theoretical framework to comprehensively and meaningfully explain reinforcement learning agents. It yields explanations with precise semantics that are not only interpretable but also mathematically justified, enabling us to identify and correct conceptual issues in prior explanations. Through illustrative examples, we show how SVERL produces useful, intuitive explanations of agent behaviour, outcomes, and predictions, which are not apparent from observing agent behaviour alone.