Explanatory Masks for Neural Network Interpretability

TL;DR

This paper introduces a method to generate explanation masks for pre-trained neural networks, helping to identify input features crucial for their predictions across various domains.

Contribution

It proposes a secondary network that produces minimal explanation masks while maintaining the original network's predictive accuracy, applicable to multiple architectures.

Findings

Effective explanation masks for CNNs, RNNs, and mixed architectures

Masks localize key input features for predictions

Method preserves accuracy with minimal explanations

Abstract

Neural network interpretability is a vital component for applications across a wide variety of domains. In such cases it is often useful to analyze a network which has already been trained for its specific purpose. In this work, we develop a method to produce explanation masks for pre-trained networks. The mask localizes the most important aspects of each input for prediction of the original network. Masks are created by a secondary network whose goal is to create as small an explanation as possible while still preserving the predictive accuracy of the original network. We demonstrate the applicability of our method for image classification with CNNs, sentiment analysis with RNNs, and chemical property prediction with mixed CNN/RNN architectures.