Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations

TL;DR

This paper critically examines the limitations of top-down model randomization-based sanity checks for evaluating explanations of deep neural networks, revealing they can be misleading and are insufficient for ranking explanation methods.

Contribution

It identifies key shortcomings of current randomization-based sanity checks, showing they can produce high scores with uninformative maps and are influenced by preserved activation scales.

Findings

Uninformative attribution maps can score highly in sanity checks.

Top-down randomization preserves activation scales, affecting explanation differences.

Sanity checks are inadequate for reliably ranking explanation methods.

Abstract

While the evaluation of explanations is an important step towards trustworthy models, it needs to be done carefully, and the employed metrics need to be well-understood. Specifically model randomization testing is often overestimated and regarded as a sole criterion for selecting or discarding certain explanation methods. To address shortcomings of this test, we start by observing an experimental gap in the ranking of explanation methods between randomization-based sanity checks [1] and model output faithfulness measures (e.g. [25]). We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Firstly, we show that uninformative attribution maps created with zero pixel-wise covariance easily achieve high scores in this type of checks. Secondly, we show that top-down model randomization preserves scales of forward pass activations with high probability. That is, channels with large activations have a high probility to contribute strongly to the output, even after randomization of the network on top of them. Hence, explanations after randomization can only be expected to differ to a certain extent. This explains the observed experimental gap. In summary, these results demonstrate the inadequacy of model-randomization-based sanity checks as a criterion to rank attribution methods.