Interpretability via Model Extraction

TL;DR

This paper introduces a model extraction technique that approximates complex blackbox models with interpretable models, enabling better understanding and debugging of machine learning systems used in critical decision-making.

Contribution

It presents a novel approach for interpreting complex models by extracting simpler, interpretable models that reflect the original model's statistical properties.

Findings

Effective interpretation of random forests and neural networks.

Successful debugging of models using extracted interpretable models.

Application to reinforcement learning policies.

Abstract

The ability to interpret machine learning models has become increasingly important now that machine learning is used to inform consequential decisions. We propose an approach called model extraction for interpreting complex, blackbox models. Our approach approximates the complex model using a much more interpretable model; as long as the approximation quality is good, then statistical properties of the complex model are reflected in the interpretable model. We show how model extraction can be used to understand and debug random forests and neural nets trained on several datasets from the UCI Machine Learning Repository, as well as control policies learned for several classical reinforcement learning problems.