How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps

TL;DR

This systematic review examines how saliency maps as explainable AI tools influence human performance, revealing mixed effects that depend on task type, AI accuracy, and experimental conditions, with limited impact from XAI features.

Contribution

It provides a comprehensive analysis of 68 studies on saliency maps, highlighting factors that modulate their effectiveness and guiding future research design.

Findings

Saliency maps can improve human performance but often show null or negative effects.

Benefits are more common in AI-focused tasks, especially for incorrect AI predictions.

XAI features have limited influence on human performance outcomes.

Abstract

Saliency maps can explain how deep neural networks classify images. But are they actually useful for humans? The present systematic review of 68 user studies found that while saliency maps can enhance human performance, null effects or even costs are quite common. To investigate what modulates these effects, the empirical outcomes were organised along several factors related to the human tasks, AI performance, XAI methods, images to be classified, human participants and comparison conditions. In image-focused tasks, benefits were less common than in AI-focused tasks, but the effects depended on the specific cognitive requirements. Moreover, benefits were usually restricted to incorrect AI predictions in AI-focused tasks but to correct ones in image-focused tasks. XAI-related factors had surprisingly little impact. The evidence was limited for image- and human-related factors and the effects were highly dependent on the comparison conditions. These findings may support the design of future user studies.