Building Reliable Explanations of Unreliable Neural Networks: Locally Smoothing Perspective of Model Interpretation

TL;DR

This paper introduces a new method for explaining neural network predictions reliably by leveraging local smoothing assumptions, which improves the identification of truly relevant input features and enhances explanation robustness against adversarial examples.

Contribution

The paper proposes a locally smoothing perspective for model interpretation, combining theoretical analysis and a regularization-based saliency map method to improve explanation reliability.

Findings

Saliency maps effectively identify relevant features in adversarial examples

Proposed method outperforms previous explanation techniques

Method reliably captures features relevant to model output

Abstract

We present a novel method for reliably explaining the predictions of neural networks. We consider an explanation reliable if it identifies input features relevant to the model output by considering the input and the neighboring data points. Our method is built on top of the assumption of smooth landscape in a loss function of the model prediction: locally consistent loss and gradient profile. A theoretical analysis established in this study suggests that those locally smooth model explanations are learned using a batch of noisy copies of the input with the L1 regularization for a saliency map. Extensive experiments support the analysis results, revealing that the proposed saliency maps retrieve the original classes of adversarial examples crafted against both naturally and adversarially trained models, significantly outperforming previous methods. We further demonstrated that such good performance results from the learning capability of this method to identify input features that are truly relevant to the model output of the input and the neighboring data points, fulfilling the requirements of a reliable explanation.