DeltaSHAP: Explaining Prediction Evolutions in Online Patient Monitoring with Shapley Values

TL;DR

DeltaSHAP is a real-time explainability method for online patient monitoring that adapts Shapley values to capture feature contributions to prediction changes over time, improving clinical interpretability and efficiency.

Contribution

It introduces DeltaSHAP, a novel temporal Shapley value-based algorithm tailored for explaining prediction evolutions in clinical time series.

Findings

Outperforms existing XAI methods in explanation quality by 62%

Reduces computation time by 33% on MIMIC-III benchmark

Effectively explains feature contributions to prediction changes in real time

Abstract

This study proposes DeltaSHAP, a novel explainable artificial intelligence (XAI) algorithm specifically designed for online patient monitoring systems. In clinical environments, discovering the causes driving patient risk evolution is critical for timely intervention, yet existing XAI methods fail to address the unique requirements of clinical time series explanation tasks. To this end, DeltaSHAP addresses three key clinical needs: explaining the changes in the consecutive predictions rather than isolated prediction scores, providing both magnitude and direction of feature attributions, and delivering these insights in real time. By adapting Shapley values to temporal settings, our approach accurately captures feature coalition effects. It further attributes prediction changes using only the actually observed feature combinations, making it efficient and practical for time-sensitive clinical applications. We also introduce new evaluation metrics to evaluate the faithfulness of the attributions for online time series, and demonstrate through experiments on online patient monitoring tasks that DeltaSHAP outperforms state-of-the-art XAI methods in both explanation quality as 62% and computational efficiency as 33% time reduction on the MIMIC-III decompensation benchmark. We release our code at https://github.com/AITRICS/DeltaSHAP.