Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models

TL;DR

This paper introduces a critic-driven Voronoi quantization method for distilling deep RL policies into interpretable models, balancing performance and interpretability by leveraging the critic network.

Contribution

It proposes a novel, model-agnostic Voronoi-based partitioning technique that uses the critic to guide the creation of simple, interpretable policies from complex RL models.

Findings

Successfully distills policies with a small set of linear functions

Outperforms traditional distillation in balancing interpretability and performance

Validated on several well-known benchmarks

Abstract

Despite many successful attempts at explaining Deep Reinforcement Learning policies using distillation, it remains difficult to balance the performance-interpretability trade-off and select a fitting surrogate model. In addition to this, traditional distillation only minimizes the distance between the behavior of the original and the surrogate policy while other RL-specific components such as action value are disregarded. To solve this, we introduce a new model-agnostic method called Critic-Driven Voronoi State Partitioning, which partitions a black box control policy into regions where a simple class of model can be optimized using gradient descent. By exploiting the critic value network of the original policy, we iteratively introduce new subpolicies in regions with insufficient value, standing in for a measure of policy complexity. The partitioning, a Voronoi quantizer, uses nearest neighbor lookups to assign a linear function to each point in the state space resulting in a cell-like diagram. We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.