Exploring How Anomalous Model Input and Output Alerts Affect Decision-Making in Healthcare

TL;DR

This study investigates how anomaly alerts in AI systems influence trust and decision-making in healthcare, revealing that while some alerts are desired, they do not necessarily improve accuracy or user experience.

Contribution

First exploration of how different anomaly alerts impact user trust and decision-making in AI-assisted healthcare settings.

Findings

Non-radiologists desire all four anomaly alerts.

Both radiologists and non-radiologists want high-confidence alerts.

Alerts did not improve radiologists' accuracy or experience.

Abstract

An important goal in the field of human-AI interaction is to help users more appropriately trust AI systems' decisions. A situation in which the user may particularly benefit from more appropriate trust is when the AI receives anomalous input or provides anomalous output. To the best of our knowledge, this is the first work towards understanding how anomaly alerts may contribute to appropriate trust of AI. In a formative mixed-methods study with 4 radiologists and 4 other physicians, we explore how AI alerts for anomalous input, very high and low confidence, and anomalous saliency-map explanations affect users' experience with mockups of an AI clinical decision support system (CDSS) for evaluating chest x-rays for pneumonia. We find evidence suggesting that the four anomaly alerts are desired by non-radiologists, and the high-confidence alerts are desired by both radiologists and non-radiologists. In a follow-up user study, we investigate how high- and low-confidence alerts affect the accuracy and thus appropriate trust of 33 radiologists working with AI CDSS mockups. We observe that these alerts do not improve users' accuracy or experience and discuss potential reasons why.