Multi-modal Evidential Fusion Network for Trustworthy PET/CT Tumor Segmentation

TL;DR

This paper introduces a novel multi-modal evidential fusion network that effectively integrates PET and CT images for more accurate and trustworthy tumor segmentation, while providing uncertainty estimates to aid clinical decision-making.

Contribution

The proposed MEFN combines cross-modal feature learning and trustworthiness fusion with uncertainty calibration, advancing tumor segmentation by handling modality uncertainty and improving robustness.

Findings

Outperforms state-of-the-art methods with over 3% DSC improvement.

Effectively models and utilizes modality uncertainty in segmentation.

Provides credible uncertainty estimates to support clinical decisions.

Abstract

Accurate tumor segmentation in PET/CT images is crucial for computer-aided cancer diagnosis and treatment. The primary challenge lies in effectively integrating the complementary information from PET and CT images. In clinical settings, the quality of PET and CT images often varies significantly, leading to uncertainty in the modality information extracted by networks. To address this challenge, we propose a novel Multi-modal Evidential Fusion Network (MEFN), which consists of two core stages: Cross-Modal Feature Learning (CFL) and Multi-modal Trustworthy Fusion (MTF). The CFL stage aligns features across different modalities and learns more robust feature representations, thereby alleviating the negative effects of domain gap. The MTF stage utilizes mutual attention mechanisms and an uncertainty calibrator to fuse modality features based on modality uncertainty and then fuse the segmentation results under the guidance of Dempster-Shafer Theory. Besides, a new uncertainty perceptual loss is introduced to force the model focusing on uncertain features and hence improve its ability to extract trusted modality information. Extensive comparative experiments are conducted on two publicly available PET/CT datasets to evaluate the performance of our proposed method whose results demonstrate that our MEFN significantly outperforms state-of-the-art methods with improvements of 3.10% and 3.23% in DSC scores on the AutoPET dataset and the Hecktor dataset, respectively. More importantly, our model can provide radiologists with credible uncertainty of the segmentation results for their decision in accepting or rejecting the automatic segmentation results, which is particularly important for clinical applications. Our code will be available at https://github.com/QPaws/MEFN.