"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agents

TL;DR

This paper introduces a novel algorithm that extracts interpretable policies from deep reinforcement learning agents, supported by theoretical guarantees, enabling understanding of complex policies without sacrificing performance.

Contribution

The paper presents a new method to derive interpretable policies from DRL agents using advantage functions, trained on existing experience, with theoretical support and empirical validation.

Findings

Successfully extracts interpretable policies in control environments.

Effectively captures expert behavior in financial trading scenarios.

Provides theoretical guarantees for the extracted policies.

Abstract

Recent advances in Reinforcement Learning (RL) largely benefit from the inclusion of Deep Neural Networks, boosting the number of novel approaches proposed in the field of Deep Reinforcement Learning (DRL). These techniques demonstrate the ability to tackle complex games such as Atari, Go, and other real-world applications, including financial trading. Nevertheless, a significant challenge emerges from the lack of interpretability, particularly when attempting to comprehend the underlying patterns learned, the relative importance of the state features, and how they are integrated to generate the policy's output. For this reason, in mission-critical and real-world settings, it is often preferred to deploy a simpler and more interpretable algorithm, although at the cost of performance. In this paper, we propose a novel algorithm, supported by theoretical guarantees, that can extract an interpretable policy (e.g., a linear policy) without disregarding the peculiarities of expert behavior. This result is obtained by considering the advantage function, which includes information about why an action is superior to the others. In contrast to previous works, our approach enables the training of an interpretable policy using previously collected experience. The proposed algorithm is empirically evaluated on classic control environments and on a financial trading scenario, demonstrating its ability to extract meaningful information from complex expert policies.