On The Coherence of Quantitative Evaluation of Visual Explanations

TL;DR

This paper critically examines the reliability and consistency of current evaluation methods for visual explanations of neural network predictions, highlighting issues of incoherence and the influence of explanation characteristics.

Contribution

It provides a comprehensive empirical study on the coherence of evaluation metrics for visual explanations using ImageNet data, including sanity checks and analysis of explanation features.

Findings

Some evaluation methods lack coherence in their grading.

Explanation characteristics like sparsity significantly affect evaluation outcomes.

Certain evaluation metrics may not reliably assess explanation quality.

Abstract

Recent years have shown an increased development of methods for justifying the predictions of neural networks through visual explanations. These explanations usually take the form of heatmaps which assign a saliency (or relevance) value to each pixel of the input image that expresses how relevant the pixel is for the prediction of a label. Complementing this development, evaluation methods have been proposed to assess the "goodness" of such explanations. On the one hand, some of these methods rely on synthetic datasets. However, this introduces the weakness of having limited guarantees regarding their applicability on more realistic settings. On the other hand, some methods rely on metrics for objective evaluation. However the level to which some of these evaluation methods perform with respect to each other is uncertain. Taking this into account, we conduct a comprehensive study on a subset of the ImageNet-1k validation set where we evaluate a number of different commonly-used explanation methods following a set of evaluation methods. We complement our study with sanity checks on the studied evaluation methods as a means to investigate their reliability and the impact of characteristics of the explanations on the evaluation methods. Results of our study suggest that there is a lack of coherency on the grading provided by some of the considered evaluation methods. Moreover, we have identified some characteristics of the explanations, e.g. sparsity, which can have a significant effect on the performance.