SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation

TL;DR

SwinMM introduces a multi-view, self-supervised learning approach using Swin Transformers to improve 3D medical image segmentation accuracy and data efficiency, especially when limited pre-training data is available.

Contribution

The paper proposes a novel masked multi-view encoder and cross-view decoder framework that leverages diverse proxy tasks and mutual learning for enhanced medical image segmentation.

Findings

Outperforms previous state-of-the-art methods like Swin UNETR.

Improves segmentation accuracy and data efficiency.

Effective multi-view information integration demonstrated.

Abstract

Recent advancements in large-scale Vision Transformers have made significant strides in improving pre-trained models for medical image segmentation. However, these methods face a notable challenge in acquiring a substantial amount of pre-training data, particularly within the medical field. To address this limitation, we present Masked Multi-view with Swin Transformers (SwinMM), a novel multi-view pipeline for enabling accurate and data-efficient self-supervised medical image analysis. Our strategy harnesses the potential of multi-view information by incorporating two principal components. In the pre-training phase, we deploy a masked multi-view encoder devised to concurrently train masked multi-view observations through a range of diverse proxy tasks. These tasks span image reconstruction, rotation, contrastive learning, and a novel task that employs a mutual learning paradigm. This new task capitalizes on the consistency between predictions from various perspectives, enabling the extraction of hidden multi-view information from 3D medical data. In the fine-tuning stage, a cross-view decoder is developed to aggregate the multi-view information through a cross-attention block. Compared with the previous state-of-the-art self-supervised learning method Swin UNETR, SwinMM demonstrates a notable advantage on several medical image segmentation tasks. It allows for a smooth integration of multi-view information, significantly boosting both the accuracy and data-efficiency of the model. Code and models are available at https://github.com/UCSC-VLAA/SwinMM/.