U-Harmony: Enhancing Joint Training for Segmentation Models with Universal Harmonization

TL;DR

U-Harmony introduces a joint training method that normalizes and denormalizes features to enable a single segmentation model to learn from diverse, heterogeneous medical datasets, improving robustness and adaptability across modalities and institutions.

Contribution

The paper presents U-Harmony, a novel joint training approach with a domain-gated head that effectively handles heterogeneous datasets and supports universal modality adaptation in medical image segmentation.

Findings

Outperforms existing methods on cross-institutional brain lesion datasets.

Establishes a new benchmark for robust 3D medical segmentation.

Demonstrates effective learning across multiple modalities and domains.

Abstract

In clinical practice, medical segmentation datasets are often limited and heterogeneous, with variations in modalities, protocols, and anatomical targets across institutions. Existing deep learning models struggle to jointly learn from such diverse data, often sacrificing either generalization or domain-specific knowledge. To overcome these challenges, we propose a joint training method called Universal Harmonization (U-Harmony), which can be integrated into deep learning-based architectures with a domain-gated head, enabling a single segmentation model to learn from heterogeneous datasets simultaneously. By integrating U-Harmony, our approach sequentially normalizes and then denormalizes feature distributions to mitigate domain-specific variations while preserving original dataset-specific knowledge. More appealingly, our framework also supports universal modality adaptation, allowing the seamless learning of new imaging modalities and anatomical classes. Extensive experiments on cross-institutional brain lesion datasets demonstrate the effectiveness of our approach, establishing a new benchmark for robust and adaptable 3D medical image segmentation models in real-world clinical settings.