Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees

TL;DR

This paper introduces Linear Model U-trees (LMUTs), a novel method to interpret deep reinforcement learning Q functions by mimicking neural network predictions with transparent, rule-extracting tree models.

Contribution

It develops the first mimic learning framework for Q functions in DRL using LMUTs, enabling interpretability and analysis of neural network policies.

Findings

LMUTs outperform baseline methods in mimicking Q functions

LMUTs facilitate understanding of feature influence and rule extraction

Empirical results demonstrate effective approximation of neural network predictions

Abstract

Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the RL problems, but often remains unexamined and uninterpreted. To our knowledge, this work develops the first mimic learning framework for Q functions in DRL. We introduce Linear Model U-trees (LMUTs) to approximate neural network predictions. An LMUT is learned using a novel on-line algorithm that is well-suited for an active play setting, where the mimic learner observes an ongoing interaction between the neural net and the environment. Empirical evaluation shows that an LMUT mimics a Q function substantially better than five baseline methods. The transparent tree structure of an LMUT facilitates understanding the network's learned knowledge by analyzing feature influence, extracting rules, and highlighting the super-pixels in image inputs.