A Human-Grounded Evaluation of SHAP for Alert Processing

TL;DR

This study evaluates whether SHAP explanations aid human experts in assessing machine learning alerts, finding that while explanations influence decision processes, they do not significantly improve alert processing performance.

Contribution

It provides the first human-grounded evaluation of SHAP in alert processing, highlighting its impact on decision-making but not on performance metrics.

Findings

SHAP explanations influence decision-making processes.

No significant performance difference with or without SHAP explanations.

Model confidence remains a primary evidence source for experts.

Abstract

In the past years, many new explanation methods have been proposed to achieve interpretability of machine learning predictions. However, the utility of these methods in practical applications has not been researched extensively. In this paper we present the results of a human-grounded evaluation of SHAP, an explanation method that has been well-received in the XAI and related communities. In particular, we study whether this local model-agnostic explanation method can be useful for real human domain experts to assess the correctness of positive predictions, i.e. alerts generated by a classifier. We performed experimentation with three different groups of participants (159 in total), who had basic knowledge of explainable machine learning. We performed a qualitative analysis of recorded reflections of experiment participants performing alert processing with and without SHAP information. The results suggest that the SHAP explanations do impact the decision-making process, although the model's confidence score remains to be a leading source of evidence. We statistically test whether there is a significant difference in task utility metrics between tasks for which an explanation was available and tasks in which it was not provided. As opposed to common intuitions, we did not find a significant difference in alert processing performance when a SHAP explanation is available compared to when it is not.