Explainable AI in Healthcare: to Explain, to Predict, or to Describe?

TL;DR

This paper critically examines the capabilities and limitations of explainable AI in healthcare, emphasizing that current methods mainly describe model behavior without providing causal explanations, which limits their practical utility.

Contribution

It introduces the Describe-Predict-Explain framework to clarify the roles of explainable AI and highlights the gap in causal explanations within current methods.

Findings

Explainable AI effectively describes model behavior.

Current methods lack causal, mechanistic explanations.

Limitations hinder actionable insights and validation.

Abstract

Explainable Artificial Intelligence (AI) methods are designed to provide information about how AI-based models make predictions. In healthcare, there is a widespread expectation that these methods will provide relevant and accurate information about a model's inner-workings to different stakeholders (ranging from patients and healthcare providers to AI and medical guideline developers). This is a challenging endeavour since what qualifies as relevant information may differ greatly depending on the stakeholder. For many stakeholders, relevant explanations are causal in nature, yet, explainable AI methods are often not able to deliver this information. Using the Describe-Predict-Explain framework, we argue that Explainable AI methods are good descriptive tools, as they may help to describe how a model works but are limited in their ability to explain why a model works in terms of true underlying biological mechanisms and cause-and-effect relations. This limits the suitability of explainable AI methods to provide actionable advice to patients or to judge the face validity of AI-based models.