VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty in Neural Network Interpretability

TL;DR

This paper introduces VOICE, a method to visualize and quantify the predictive uncertainty of gradient-based explanations for neural networks, revealing insights into explanation trustworthiness and uncertainty behavior.

Contribution

It provides a theoretical analysis of existing evaluation strategies and proposes a plug-in approach to measure residual uncertainty in explanations.

Findings

Uncertainty varies across images, networks, and explanations.

Incorrect predictions often have high explanation uncertainty.

Uncertainty metrics align with epistemic uncertainty.

Abstract

In this paper, we visualize and quantify the predictive uncertainty of gradient-based post hoc visual explanations for neural networks. Predictive uncertainty refers to the variability in the network predictions under perturbations to the input. Visual post hoc explainability techniques highlight features within an image to justify a network's prediction. We theoretically show that existing evaluation strategies of visual explanatory techniques partially reduce the predictive uncertainty of neural networks. This analysis allows us to construct a plug in approach to visualize and quantify the remaining predictive uncertainty of any gradient-based explanatory technique. We show that every image, network, prediction, and explanatory technique has a unique uncertainty. The proposed uncertainty visualization and quantification yields two key observations. Firstly, oftentimes under incorrect predictions, explanatory techniques are uncertain about the same features that they are attributing the predictions to, thereby reducing the trustworthiness of the explanation. Secondly, objective metrics of an explanation's uncertainty, empirically behave similarly to epistemic uncertainty. We support these observations on two datasets, four explanatory techniques, and six neural network architectures. The code is available at https://github.com/olivesgatech/VOICE-Uncertainty.