Evaluation of Popular XAI Applied to Clinical Prediction Models: Can They be Trusted?

TL;DR

This study benchmarks two popular XAI methods in healthcare, assessing their practicality, trustworthiness, and impact on clinical workflows using EMR data, revealing limitations and potential benefits for clinical decision support.

Contribution

It provides the first benchmarking of XAI methods in clinical risk prediction, evaluating their coherence, impact, and consistency in real-world healthcare datasets.

Findings

XAI methods show limitations in clinical contexts

Explanations can influence clinical decision-making

Benchmarking reveals gaps in current XAI approaches

Abstract

The absence of transparency and explainability hinders the clinical adoption of Machine learning (ML) algorithms. Although various methods of explainable artificial intelligence (XAI) have been suggested, there is a lack of literature that delves into their practicality and assesses them based on criteria that could foster trust in clinical environments. To address this gap this study evaluates two popular XAI methods used for explaining predictive models in the healthcare context in terms of whether they (i) generate domain-appropriate representation, i.e. coherent with respect to the application task, (ii) impact clinical workflow and (iii) are consistent. To that end, explanations generated at the cohort and patient levels were analysed. The paper reports the first benchmarking of the XAI methods applied to risk prediction models obtained by evaluating the concordance between generated explanations and the trigger of a future clinical deterioration episode recorded by the data collection system. We carried out an analysis using two Electronic Medical Records (EMR) datasets sourced from Australian major hospitals. The findings underscore the limitations of state-of-the-art XAI methods in the clinical context and their potential benefits. We discuss these limitations and contribute to the theoretical development of trustworthy XAI solutions where clinical decision support guides the choice of intervention by suggesting the pattern or drivers for clinical deterioration in the future.