Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks

TL;DR

This survey reviews the development of Self-eXplainable AI in medical image analysis, emphasizing models that inherently generate explanations during training to improve transparency, trustworthiness, and accountability in healthcare applications.

Contribution

It provides a comprehensive overview of S-XAI methods for medical imaging, covering input, model, and output explainability, and discusses evaluation metrics, challenges, and future research directions.

Findings

Over 200 papers reviewed across multiple modalities and applications

Identified key techniques: explainable feature engineering, attention, concepts, prototypes

Outlined evaluation criteria and future challenges for S-XAI in medicine

Abstract

The increasing demand for transparent and reliable models, particularly in high-stakes decision-making areas such as medical image analysis, has led to the emergence of eXplainable Artificial Intelligence (XAI). Post-hoc XAI techniques, which aim to explain black-box models after training, have raised concerns about their fidelity to model predictions. In contrast, Self-eXplainable AI (S-XAI) offers a compelling alternative by incorporating explainability directly into the training process of deep learning models. This approach allows models to generate inherent explanations that are closely aligned with their internal decision-making processes, enhancing transparency and supporting the trustworthiness, robustness, and accountability of AI systems in real-world medical applications. To facilitate the development of S-XAI methods for medical image analysis, this survey presents a comprehensive review across various image modalities and clinical applications. It covers more than 200 papers from three key perspectives: 1) input explainability through the integration of explainable feature engineering and knowledge graph, 2) model explainability via attention-based learning, concept-based learning, and prototype-based learning, and 3) output explainability by providing textual and counterfactual explanations. This paper also outlines desired characteristics of explainability and evaluation methods for assessing explanation quality, while discussing major challenges and future research directions in developing S-XAI for medical image analysis.