Learning to Explain: An Information-Theoretic Perspective on Model Interpretation

TL;DR

This paper proposes a novel instancewise feature selection method for model interpretation, maximizing mutual information between selected features and responses, with demonstrated effectiveness on synthetic and real datasets.

Contribution

It introduces an information-theoretic approach to feature selection for model explanation, including a variational approximation for mutual information.

Findings

Effective on synthetic datasets

Validated with human evaluation

Outperforms baseline methods

Abstract

We introduce instancewise feature selection as a methodology for model interpretation. Our method is based on learning a function to extract a subset of features that are most informative for each given example. This feature selector is trained to maximize the mutual information between selected features and the response variable, where the conditional distribution of the response variable given the input is the model to be explained. We develop an efficient variational approximation to the mutual information, and show the effectiveness of our method on a variety of synthetic and real data sets using both quantitative metrics and human evaluation.