PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates

TL;DR

PASSION introduces a novel approach for incomplete multi-modal medical image segmentation that effectively handles imbalanced missing rates by balancing modality contributions during training, improving performance and robustness.

Contribution

The paper formulates a new setting for incomplete multi-modal segmentation with imbalanced missing rates and proposes PASSION, a self-distillation method with preference-aware regularization, for improved modality balancing.

Findings

PASSION outperforms existing methods on two public datasets.

PASSION improves segmentation accuracy with imbalanced missing modalities.

PASSION enhances model robustness as a plug-and-play module.

Abstract

Incomplete multi-modal image segmentation is a fundamental task in medical imaging to refine deployment efficiency when only partial modalities are available. However, the common practice that complete-modality data is visible during model training is far from realistic, as modalities can have imbalanced missing rates in clinical scenarios. In this paper, we, for the first time, formulate such a challenging setting and propose Preference-Aware Self-diStillatION (PASSION) for incomplete multi-modal medical image segmentation under imbalanced missing rates. Specifically, we first construct pixel-wise and semantic-wise self-distillation to balance the optimization objective of each modality. Then, we define relative preference to evaluate the dominance of each modality during training, based on which to design task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate the superiority of PASSION against existing approaches for modality balancing. More importantly, PASSION is validated to work as a plug-and-play module for consistent performance improvement across different backbones. Code is available at https://github.com/Jun-Jie-Shi/PASSION.