Auditor models to suppress poor artificial intelligence predictions can improve human-artificial intelligence collaborative performance

TL;DR

This paper shows that suppressing unreliable AI predictions can improve fairness and performance in human-AI healthcare decision-making.

Contribution

The study introduces auditor models to suppress poor AI predictions, improving collaborative fairness and performance in healthcare decisions.

Findings

Suppression of poor ML predictions improves collaborative performance when ML outperforms humans.

Incorporating uncertainty quantification enhances suppression effectiveness.

Suppression maintains fairness and avoids degrading performance when ML is less accurate.

Abstract

Healthcare decisions are increasingly made with the assistance of machine learning (ML). ML has been known to have unfairness—inconsistent outcomes across subpopulations. Clinicians interacting with these systems can perpetuate such unfairness by overreliance. Recent work exploring ML suppression—silencing predictions based on auditing the ML—shows promise in mitigating performance issues originating from overreliance. This study aims to evaluate the impact of suppression on collaboration fairness and evaluate ML uncertainty as desiderata to audit the ML.

We used data from the Vanderbilt University Medical Center electronic health record (n = 58 817) and the MIMIC-IV-ED dataset (n = 363 145) to predict likelihood of death or intensive care unit transfer and likelihood of 30-day readmission using gradient-boosted trees and an artificially high-performing oracle model. We derived clinician decisions directly from the dataset and simulated clinician acceptance of ML predictions based on previous empirical work on acceptance of clinical decision support alerts. We measured performance as area under the receiver operating characteristic curve and algorithmic fairness using absolute averaged odds difference.

When the ML outperforms humans, suppression outperforms the human alone (P < 8.2 × 10−6) and at least does not degrade fairness. When the human outperforms the ML, the human is either fairer than suppression (P < 8.2 × 10−4) or there is no statistically significant difference in fairness. Incorporating uncertainty quantification into suppression approaches can improve performance.

Suppression of poor-quality ML predictions through an auditor model shows promise in improving collaborative human-AI performance and fairness.

Figures