Cross-dimensional transfer learning in medical image segmentation with deep learning

TL;DR

This paper presents a novel transfer learning approach that leverages 2D pre-trained networks to improve medical image segmentation across 2D and 3D modalities, achieving state-of-the-art results on multiple benchmarks.

Contribution

The authors introduce architectures that embed 2D pre-trained encoders into higher-dimensional networks and expand 2D segmentation networks into 3D, enabling effective cross-dimensional transfer learning in medical imaging.

Findings

Ranked first on the CAMUS challenge for echo-cardiographic segmentation.

Outperformed existing 2D methods on CHAOS challenge for MR and CT images.

Achieved promising results on BraTS 2022 for brain tumor segmentation.

Abstract

Over the last decade, convolutional neural networks have emerged and advanced the state-of-the-art in various image analysis and computer vision applications. The performance of 2D image classification networks is constantly improving and being trained on databases made of millions of natural images. However, progress in medical image analysis has been hindered by limited annotated data and acquisition constraints. These limitations are even more pronounced given the volumetry of medical imaging data. In this paper, we introduce an efficient way to transfer the efficiency of a 2D classification network trained on natural images to 2D, 3D uni- and multi-modal medical image segmentation applications. In this direction, we designed novel architectures based on two key principles: weight transfer by embedding a 2D pre-trained encoder into a higher dimensional U-Net, and dimensional transfer by expanding a 2D segmentation network into a higher dimension one. The proposed networks were tested on benchmarks comprising different modalities: MR, CT, and ultrasound images. Our 2D network ranked first on the CAMUS challenge dedicated to echo-cardiographic data segmentation and surpassed the state-of-the-art. Regarding 2D/3D MR and CT abdominal images from the CHAOS challenge, our approach largely outperformed the other 2D-based methods described in the challenge paper on Dice, RAVD, ASSD, and MSSD scores and ranked third on the online evaluation platform. Our 3D network applied to the BraTS 2022 competition also achieved promising results, reaching an average Dice score of 91.69% (91.22%) for the whole tumor, 83.23% (84.77%) for the tumor core, and 81.75% (83.88%) for enhanced tumor using the approach based on weight (dimensional) transfer. Experimental and qualitative results illustrate the effectiveness of our methods for multi-dimensional medical image segmentation.