Saliency Maps are Ambiguous: Analysis of Logical Relations on First and Second Order Attributions

TL;DR

This paper critically examines the limitations of saliency map methods in explaining model decisions, revealing their failure to capture essential classification information across various scenarios and proposing new evaluation approaches.

Contribution

It extends prior analysis of saliency methods using the ANDOR logical dataset framework and introduces the Global Coherence Representation for improved evaluation.

Findings

All analyzed saliency methods fail to capture complete classification information.

Masking-based evaluations can introduce biases in complex data.

The Global Coherence Representation enables more accurate input omission analysis.

Abstract

Recent work uncovered potential flaws in \eg attribution or heatmap based saliency methods. A typical flaw is a confirmations bias, where the scores are compared to human expectation. Since measuring the quality of saliency methods is hard due to missing ground truth model reasoning, finding general limitations is also hard. This is further complicated, because masking-based evaluation on complex data can easily introduce a bias, as most methods cannot fully ignore inputs. In this work, we extend our previous analysis on the logical dataset framework ANDOR, where we showed that all analysed saliency methods fail to grasp all needed classification information for all possible scenarios. Specifically, this paper extends our previous work using analysis on more datasets, in order to better understand in which scenarios the saliency methods fail. Further, we apply the Global Coherence Representation as an additional evaluation method in order to enable actual input omission.