Causal policy ranking

TL;DR

This paper introduces a black-box causal method to rank decisions in reinforcement learning policies based on their direct impact on rewards, enhancing interpretability of complex RL policies.

Contribution

It proposes a novel counterfactual reasoning approach for causal policy ranking in RL, providing a new way to interpret decision importance.

Findings

Causal ranking correlates with reward contribution.

Causal method outperforms non-causal ranking in interpretability.

Preliminary results show promise for causal interpretability in RL.

Abstract

Policies trained via reinforcement learning (RL) are often very complex even for simple tasks. In an episode with $n$ time steps, a policy will make $n$ decisions on actions to take, many of which may appear non-intuitive to the observer. Moreover, it is not clear which of these decisions directly contribute towards achieving the reward and how significant is their contribution. Given a trained policy, we propose a black-box method based on counterfactual reasoning that estimates the causal effect that these decisions have on reward attainment and ranks the decisions according to this estimate. In this preliminary work, we compare our measure against an alternative, non-causal, ranking procedure, highlight the benefits of causality-based policy ranking, and discuss potential future work integrating causal algorithms into the interpretation of RL agent policies.