Towards Better Understanding Attribution Methods

TL;DR

This paper introduces new evaluation schemes for attribution methods in neural networks, improving fairness, reliability, and systematic analysis of their faithfulness and visualizations.

Contribution

It proposes three novel evaluation schemes for attribution methods, addressing faithfulness, fairness, and visualization, and introduces a smoothing step to enhance attribution performance.

Findings

DiFull effectively distinguishes possible from impossible attributions.

Evaluating all methods on the same layers reveals performance differences.

Smoothing improves attribution accuracy for certain methods.

Abstract

Deep neural networks are very successful on many vision tasks, but hard to interpret due to their black box nature. To overcome this, various post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. Evaluating such methods is challenging since no ground truth attributions exist. We thus propose three novel evaluation schemes to more reliably measure the faithfulness of those methods, to make comparisons between them more fair, and to make visual inspection more systematic. To address faithfulness, we propose a novel evaluation setting (DiFull) in which we carefully control which parts of the input can influence the output in order to distinguish possible from impossible attributions. To address fairness, we note that different methods are applied at different layers, which skews any comparison, and so evaluate all methods on the same layers (ML-Att) and discuss how this impacts their performance on quantitative metrics. For more systematic visualizations, we propose a scheme (AggAtt) to qualitatively evaluate the methods on complete datasets. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods. Finally, we propose a post-processing smoothing step that significantly improves the performance of some attribution methods, and discuss its applicability.