The Explanation Necessity for Healthcare AI

TL;DR

This paper introduces a categorization system and mathematical framework to determine the necessity and level of explainability required for AI applications in healthcare, aiming to improve trust and decision-making transparency.

Contribution

It proposes a novel explanation necessity categorization and a mathematical formulation to guide explainability levels in medical AI applications.

Findings

A four-class categorization system for explanation necessity

A mathematical model incorporating robustness, variability, and dimensionality

Guidelines for when and how to explain AI decisions in healthcare

Abstract

Explainability is a critical factor in enhancing the trustworthiness and acceptance of artificial intelligence (AI) in healthcare, where decisions directly impact patient outcomes. Despite advancements in AI interpretability, clear guidelines on when and to what extent explanations are required in medical applications remain lacking. We propose a novel categorization system comprising four classes of explanation necessity (self-explainable, semi-explainable, non-explainable, and new-patterns discovery), guiding the required level of explanation; whether local (patient or sample level), global (cohort or dataset level), or both. To support this system, we introduce a mathematical formulation that incorporates three key factors: (i) robustness of the evaluation protocol, (ii) variability of expert observations, and (iii) representation dimensionality of the application. This framework provides a practical tool for researchers to determine the appropriate depth of explainability needed, addressing the critical question: When does an AI medical application need to be explained, and at what level of detail?