Focus! Rating XAI Methods and Finding Biases

TL;DR

This paper introduces the Focus score, an evaluation metric for feature attribution methods in image recognition, assessing their coherency and bias, and compares popular explainability techniques across CNN models.

Contribution

The paper proposes a novel evaluation score, Focus, for assessing feature attribution methods within the distribution, and demonstrates its use in comparing explainability techniques and identifying model biases.

Findings

LRP and GradCAM are consistently reliable explanations.

SmoothGrad and IG tend to produce class-agnostic explanations.

Focus effectively identifies and characterizes biases in models.

Abstract

AI explainability improves the transparency of models, making them more trustworthy. Such goals are motivated by the emergence of deep learning models, which are obscure by nature; even in the domain of images, where deep learning has succeeded the most, explainability is still poorly assessed. In the field of image recognition many feature attribution methods have been proposed with the purpose of explaining a model's behavior using visual cues. However, no metrics have been established so far to assess and select these methods objectively. In this paper we propose a consistent evaluation score for feature attribution methods -- the Focus -- designed to quantify their coherency to the task. While most previous work adds out-of-distribution noise to samples, we introduce a methodology to add noise from within the distribution. This is done through mosaics of instances from different classes, and the explanations these generate. On those, we compute a visual pseudo-precision metric, Focus. First, we show the robustness of the approach through a set of randomization experiments. Then we use Focus to compare six popular explainability techniques across several CNN architectures and classification datasets. Our results find some methods to be consistently reliable (LRP, GradCAM), while others produce class-agnostic explanations (SmoothGrad, IG). Finally we introduce another application of Focus, using it for the identification and characterization of biases found in models. This empowers bias-management tools, in another small step towards trustworthy AI.