Explainable Reinforcement Learning via Temporal Policy Decomposition

TL;DR

This paper introduces Temporal Policy Decomposition (TPD), a novel method for explaining reinforcement learning policies by decomposing value functions into expected future outcomes over time, enhancing interpretability.

Contribution

The paper proposes TPD, a new approach that explains RL actions through temporal decomposition of value functions, improving transparency and understanding of policy behavior.

Findings

TPD provides accurate, interpretable explanations of RL actions.

It clarifies the policy's future strategy and anticipated outcomes.

The method aids in aligning reward functions with human expectations.

Abstract

We investigate the explainability of Reinforcement Learning (RL) policies from a temporal perspective, focusing on the sequence of future outcomes associated with individual actions. In RL, value functions compress information about rewards collected across multiple trajectories and over an infinite horizon, allowing a compact form of knowledge representation. However, this compression obscures the temporal details inherent in sequential decision-making, presenting a key challenge for interpretability. We present Temporal Policy Decomposition (TPD), a novel explainability approach that explains individual RL actions in terms of their Expected Future Outcome (EFO). These explanations decompose generalized value functions into a sequence of EFOs, one for each time step up to a prediction horizon of interest, revealing insights into when specific outcomes are expected to occur. We leverage fixed-horizon temporal difference learning to devise an off-policy method for learning EFOs for both optimal and suboptimal actions, enabling contrastive explanations consisting of EFOs for different state-action pairs. Our experiments demonstrate that TPD generates accurate explanations that (i) clarify the policy's future strategy and anticipated trajectory for a given action and (ii) improve understanding of the reward composition, facilitating fine-tuning of the reward function to align with human expectations.