Learning Multi-Modal Brain Tumor Segmentation from Privileged Semi-Paired MRI Images with Curriculum Disentanglement Learning

TL;DR

This paper introduces a two-step curriculum disentanglement framework that leverages privileged semi-paired MRI images to improve multi-modal brain tumor segmentation, effectively capturing complementary modality information.

Contribution

The novel two-step curriculum disentanglement learning approach effectively utilizes semi-paired images for enhanced brain tumor segmentation, outperforming existing models.

Findings

Outperforms competing models on three brain tumor segmentation tasks.

Effectively extracts modality-specific style and invariant content codes.

Utilizes privileged semi-paired images to improve segmentation accuracy.

Abstract

Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor segmentation. Specifically, in the first step, we propose to conduct reconstruction and segmentation with augmented intra-modality style-consistent images. In the second step, the model jointly performs reconstruction, unsupervised/supervised translation, and segmentation for both unpaired and paired inter-modality images. A content consistency loss and a supervised translation loss are proposed to leverage complementary information from different modalities in this step. Through these two steps, our method effectively extracts modality-specific style codes describing the attenuation of tissue features and image contrast, and modality-invariant content codes containing anatomical and functional information from the input images. Experiments on three brain tumor segmentation tasks show that our model outperforms competing segmentation models based on unpaired images.