The Three Ghosts of Medical AI: Can the Black-Box Present Deliver?

TL;DR

This paper explores the challenges of using opaque, black-box machine learning models in medical AI, emphasizing the need for transparency to ensure trust, quality, and effective physician-patient communication.

Contribution

It critically reviews the limitations of black-box models in healthcare and advocates for transparency in model design and validation to improve medical AI reliability.

Findings

Opaque models lack quality assurance

Opaque models fail to elicit trust

Opacity restricts physician-patient dialogue

Abstract

Our title alludes to the three Christmas ghosts encountered by Ebenezer Scrooge in \textit{A Christmas Carol}, who guide Ebenezer through the past, present, and future of Christmas holiday events. Similarly, our article will take readers through a journey of the past, present, and future of medical AI. In doing so, we focus on the crux of modern machine learning: the reliance on powerful but intrinsically opaque models. When applied to the healthcare domain, these models fail to meet the needs for transparency that their clinician and patient end-users require. We review the implications of this failure, and argue that opaque models (1) lack quality assurance, (2) fail to elicit trust, and (3) restrict physician-patient dialogue. We then discuss how upholding transparency in all aspects of model design and model validation can help ensure the reliability of medical AI.