CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images

TL;DR

CmFNet introduces a cross-modal fusion network that leverages sparse annotations and multi-modal data to improve weakly supervised medical image segmentation, outperforming existing methods and even some fully supervised approaches.

Contribution

The paper presents a novel 3D weakly supervised segmentation method combining modality-specific and cross-modal feature learning with hybrid supervision, effectively handling sparse annotations and multi-modal data.

Findings

Outperforms state-of-the-art weakly supervised segmentation methods.

Achieves comparable or better results than fully supervised methods.

Effectively segments small tumors and anatomical structures.

Abstract

Accurate automatic medical image segmentation relies on high-quality, dense annotations, which are costly and time-consuming. Weakly supervised learning provides a more efficient alternative by leveraging sparse and coarse annotations instead of dense, precise ones. However, segmentation performance degradation and overfitting caused by sparse annotations remain key challenges. To address these issues, we propose CmFNet, a novel 3D weakly supervised cross-modal medical image segmentation approach. CmFNet consists of three main components: a modality-specific feature learning network, a cross-modal feature learning network, and a hybrid-supervised learning strategy. Specifically, the modality-specific feature learning network and the cross-modal feature learning network effectively integrate complementary information from multi-modal images, enhancing shared features across modalities to improve segmentation performance. Additionally, the hybrid-supervised learning strategy guides segmentation through scribble supervision, intra-modal regularization, and inter-modal consistency, modeling spatial and contextual relationships while promoting feature alignment. Our approach effectively mitigates overfitting, delivering robust segmentation results. It excels in segmenting both challenging small tumor regions and common anatomical structures. Extensive experiments on a clinical cross-modal nasopharyngeal carcinoma (NPC) dataset (including CT and MR imaging) and the publicly available CT Whole Abdominal Organ dataset (WORD) show that our approach outperforms state-of-the-art weakly supervised methods. In addition, our approach also outperforms fully supervised methods when full annotation is used. Our approach can facilitate clinical therapy and benefit various specialists, including physicists, radiologists, pathologists, and oncologists.