The benefits and costs of explainable artificial intelligence in visual quality control: Evidence from fault detection performance and eye movements

TL;DR

This study investigates how explainable AI impacts human visual quality control, showing that XAI can improve speed but may also lead to errors or reduced effort depending on AI accuracy and highlight correctness.

Contribution

The paper provides empirical evidence on the benefits and costs of XAI in visual inspection tasks, highlighting the importance of AI accuracy and highlight placement on human decision-making.

Findings

XAI speeds up fault detection performance.

XAI benefits depend on AI and highlight accuracy.

XAI can reduce human effort and attention to details.

Abstract

Visual inspection tasks often require humans to cooperate with AI-based image classifiers. To enhance this cooperation, explainable artificial intelligence (XAI) can highlight those image areas that have contributed to an AI decision. However, the literature on visual cueing suggests that such XAI support might come with costs of its own. To better understand how the benefits and cost of XAI depend on the accuracy of AI classifications and XAI highlights, we conducted two experiments that simulated visual quality control in a chocolate factory. Participants had to decide whether chocolate moulds contained faulty bars or not, and were always informed whether the AI had classified the mould as faulty or not. In half of the experiment, they saw additional XAI highlights that justified this classification. While XAI speeded up performance, its effects on error rates were highly dependent on (X)AI accuracy. XAI benefits were observed when the system correctly detected and highlighted the fault, but XAI costs were evident for misplaced highlights that marked an intact area while the actual fault was located elsewhere. Eye movement analyses indicated that participants spent less time searching the rest of the mould and thus looked at the fault less often. However, we also observed large interindividual differences. Taken together, the results suggest that despite its potentials, XAI can discourage people from investing effort into their own information analysis.