Soft Tissue Sarcoma Co-Segmentation in Combined MRI and PET/CT Data

TL;DR

This paper introduces a novel multimodal co-segmentation approach for soft tissue sarcoma that leverages MRI and PET/CT data to improve tumor delineation on each modality, addressing the limitations of previous fusion methods.

Contribution

It develops a modality-specific co-segmentation model with dedicated encoder-decoder branches and densely connected layers, enhancing tumor segmentation accuracy across modalities.

Findings

Multimodal co-segmentation outperforms single-modality models.

Different input modalities and fusion strategies significantly impact segmentation quality.

The approach is validated on public soft tissue sarcoma datasets with positive results.

Abstract

Tumor segmentation in multimodal medical images has seen a growing trend towards deep learning based methods. Typically, studies dealing with this topic fuse multimodal image data to improve the tumor segmentation contour for a single imaging modality. However, they do not take into account that tumor characteristics are emphasized differently by each modality, which affects the tumor delineation. Thus, the tumor segmentation is modality- and task-dependent. This is especially the case for soft tissue sarcomas, where, due to necrotic tumor tissue, the segmentation differs vastly. Closing this gap, we develop a modalityspecific sarcoma segmentation model that utilizes multimodal image data to improve the tumor delineation on each individual modality. We propose a simultaneous co-segmentation method, which enables multimodal feature learning through modality-specific encoder and decoder branches, and the use of resource-effcient densely connected convolutional layers. We further conduct experiments to analyze how different input modalities and encoder-decoder fusion strategies affect the segmentation result. We demonstrate the effectiveness of our approach on public soft tissue sarcoma data, which comprises MRI (T1 and T2 sequence) and PET/CT scans. The results show that our multimodal co-segmentation model provides better modality-specific tumor segmentation than models using only the PET or MRI (T1 and T2) scan as input.