CLoE: Expert Consistency Learning for Missing Modality Segmentation

TL;DR

This paper introduces CLoE, a novel framework for multimodal medical image segmentation that maintains high accuracy despite missing modalities by enforcing expert prediction consistency and reliability-aware fusion.

Contribution

The paper proposes a dual-branch expert consistency learning framework that improves robustness and reliability in missing-modality segmentation tasks, outperforming existing methods.

Findings

Outperforms state-of-the-art methods on BraTS 2020 and MSD Prostate datasets.

Enhances robustness and generalization across different datasets.

Improves segmentation accuracy on clinically critical structures.

Abstract

Multimodal medical image segmentation often faces missing modalities at inference, which induces disagreement among modality experts and makes fusion unstable, particularly on small foreground structures. We propose Consistency Learning of Experts (CLoE), a consistency-driven framework for missing-modality segmentation that preserves strong performance when all modalities are available. CLoE formulates robustness as decision-level expert consistency control and introduces a dual-branch Expert Consistency Learning objective. Modality Expert Consistency enforces global agreement among expert predictions to reduce case-wise drift under partial inputs, while Region Expert Consistency emphasizes agreement on clinically critical foreground regions to avoid background-dominated regularization. We further map consistency scores to modality reliability weights using a lightweight gating network, enabling reliability-aware feature recalibration before fusion. Extensive experiments on BraTS 2020 and MSD Prostate demonstrate that CLoE outperforms state-of-the-art methods in incomplete multimodal segmentation, while exhibiting strong cross-dataset generalization and improving robustness on clinically critical structures.