Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation

TL;DR

This paper introduces a logical reasoning-based testing framework for evaluating saliency methods, revealing how these methods encode class-relevant information and highlighting limitations of current evaluation practices.

Contribution

It proposes a novel evaluation approach using logical datasets and new metrics to analyze saliency methods' treatment of information, addressing biases in existing evaluation methods.

Findings

Saliency methods encode classification information in score orderings.

Current evaluation practices may be biased and insufficient.

Logical datasets reveal how methods handle complementary and redundant info.

Abstract

With their increase in performance, neural network architectures also become more complex, necessitating explainability. Therefore, many new and improved methods are currently emerging, which often generate so-called saliency maps in order to improve interpretability. Those methods are often evaluated by visual expectations, yet this typically leads towards a confirmation bias. Due to a lack of a general metric for explanation quality, non-accessible ground truth data about the model's reasoning and the large amount of involved assumptions, multiple works claim to find flaws in those methods. However, this often leads to unfair comparison metrics. Additionally, the complexity of most datasets (mostly images or text) is often so high, that approximating all possible explanations is not feasible. For those reasons, this paper introduces a test for saliency map evaluation: proposing controlled experiments based on all possible model reasonings over multiple simple logical datasets. Using the contained logical relationships, we aim to understand how different saliency methods treat information in different class discriminative scenarios (e.g. via complementary and redundant information). By introducing multiple new metrics, we analyse propositional logical patterns towards a non-informative attribution score baseline to find deviations of typical expectations. Our results show that saliency methods can encode classification relevant information into the ordering of saliency scores.