Failure Modes of Time Series Interpretability Algorithms for Critical Care Applications and Potential Solutions

TL;DR

This paper analyzes failure modes of interpretability algorithms in critical care time series models and proposes learnable mask-based methods to improve reliability and consistency of explanations.

Contribution

It systematically identifies failure modes of existing interpretability algorithms and introduces learnable mask-based approaches that incorporate temporal constraints for better explanations.

Findings

Gradient, Occlusion, and Permutation methods struggle with time-varying dependencies.

Learnable mask-based methods incorporate temporal continuity and label consistency.

Proposed methods offer more reliable interpretations in critical care applications.

Abstract

Interpretability plays a vital role in aligning and deploying deep learning models in critical care, especially in constantly evolving conditions that influence patient survival. However, common interpretability algorithms face unique challenges when applied to dynamic prediction tasks, where patient trajectories evolve over time. Gradient, Occlusion, and Permutation-based methods often struggle with time-varying target dependency and temporal smoothness. This work systematically analyzes these failure modes and supports learnable mask-based interpretability frameworks as alternatives, which can incorporate temporal continuity and label consistency constraints to learn feature importance over time. Here, we propose that learnable mask-based approaches for dynamic timeseries prediction problems provide more reliable and consistent interpretations for applications in critical care and similar domains.