Causal Attribution of Model Performance Gaps in Medical Imaging Under Distribution Shifts

TL;DR

This paper introduces a causal attribution framework for understanding performance drops in medical image segmentation models under distribution shifts, highlighting the distinct impacts of acquisition and annotation variability.

Contribution

It extends causal attribution methods to high-dimensional medical imaging tasks, enabling quantification of different factors' contributions to performance degradation.

Findings

Annotation protocol shifts significantly impact performance when crossing annotators.

Acquisition protocol shifts dominate when crossing imaging centers.

The framework helps prioritize targeted interventions based on specific deployment contexts.

Abstract

Deep learning models for medical image segmentation suffer significant performance drops due to distribution shifts, but the causal mechanisms behind these drops remain poorly understood. We extend causal attribution frameworks to high-dimensional segmentation tasks, quantifying how acquisition protocols and annotation variability independently contribute to performance degradation. We model the data-generating process through a causal graph and employ Shapley values to fairly attribute performance changes to individual mechanisms. Our framework addresses unique challenges in medical imaging: high-dimensional outputs, limited samples, and complex mechanism interactions. Validation on multiple sclerosis (MS) lesion segmentation across 4 centers and 7 annotators reveals context-dependent failure modes: annotation protocol shifts dominate when crossing annotators (7.4% $\pm$ 8.9% DSC attribution), while acquisition shifts dominate when crossing imaging centers (6.5% $\pm$ 9.1%). This mechanism-specific quantification enables practitioners to prioritize targeted interventions based on deployment context.