Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

TL;DR

This paper introduces Iterative Bounding MDPs (IBMDPs), a novel framework that enables learning interpretable decision tree policies using any function approximator, including neural networks, by ensuring policy interpretability through a decision tree equivalence.

Contribution

The paper proposes IBMDPs, a new MDP type that guarantees decision tree policies for the base MDPs, compatible with modern neural network training methods.

Findings

Successfully produced decision tree policies for base MDPs using IBMDPs.

Demonstrated the approach's compatibility with neural network training.

Showed empirical benefits of interpretability in reinforcement learning policies.

Abstract

Current work in explainable reinforcement learning generally produces policies in the form of a decision tree over the state space. Such policies can be used for formal safety verification, agent behavior prediction, and manual inspection of important features. However, existing approaches fit a decision tree after training or use a custom learning procedure which is not compatible with new learning techniques, such as those which use neural networks. To address this limitation, we propose a novel Markov Decision Process (MDP) type for learning decision tree policies: Iterative Bounding MDPs (IBMDPs). An IBMDP is constructed around a base MDP so each IBMDP policy is guaranteed to correspond to a decision tree policy for the base MDP when using a method-agnostic masking procedure. Because of this decision tree equivalence, any function approximator can be used during training, including a neural network, while yielding a decision tree policy for the base MDP. We present the required masking procedure as well as a modified value update step which allows IBMDPs to be solved using existing algorithms. We apply this procedure to produce IBMDP variants of recent reinforcement learning methods. We empirically show the benefits of our approach by solving IBMDPs to produce decision tree policies for the base MDPs.