Regularizing Black-box Models for Improved Interpretability

TL;DR

This paper introduces ExpO, a hybrid approach that regularizes black-box models during training to enhance explanation quality, resulting in more faithful and stable explanations without domain-specific knowledge.

Contribution

ExpO is a novel, differentiable, model-agnostic regularization method that improves post-hoc explanation quality for black-box models during training.

Findings

ExpO-regularized models produce explanations with higher fidelity.

ExpO improves explanation stability compared to baseline models.

User study confirms more useful explanations with ExpO.

Abstract

Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, whose explanation quality can be unpredictable. Our method, ExpO, is a hybridization of these approaches that regularizes a model for explanation quality at training time. Importantly, these regularizers are differentiable, model agnostic, and require no domain knowledge to define. We demonstrate that post-hoc explanations for ExpO-regularized models have better explanation quality, as measured by the common fidelity and stability metrics. We verify that improving these metrics leads to significantly more useful explanations with a user study on a realistic task.