User Trust on an Explainable AI-based Medical Diagnosis Support System

TL;DR

This study investigates how explainability in AI-based medical diagnosis systems influences radiologists' trust and reliance, highlighting the importance of causal explanations and user feedback in improving trust levels.

Contribution

The paper introduces a system providing causal explanations for diagnoses and evaluates its impact on radiologist trust using a real dataset and user feedback.

Findings

Model achieved 74.1% accuracy on CXR-Eye dataset.

Radiologists agreed with model predictions in 46.4% of cases.

Self-reported trust score was 3.2 out of 5.

Abstract

Recent research has supported that system explainability improves user trust and willingness to use medical AI for diagnostic support. In this paper, we use chest disease diagnosis based on X-Ray images as a case study to investigate user trust and reliance. Building off explainability, we propose a support system where users (radiologists) can view causal explanations for final decisions. After observing these causal explanations, users provided their opinions of the model predictions and could correct explanations if they did not agree. We measured user trust as the agreement between the model's and the radiologist's diagnosis as well as the radiologists' feedback on the model explanations. Additionally, they reported their trust in the system. We tested our model on the CXR-Eye dataset and it achieved an overall accuracy of 74.1%. However, the experts in our user study agreed with the model for only 46.4% of the cases, indicating the necessity of improving the trust. The self-reported trust score was 3.2 on a scale of 1.0 to 5.0, showing that the users tended to trust the model but the trust still needs to be enhanced.