VIViT: Variable-Input Vision Transformer Framework for 3D MR Image Segmentation

TL;DR

VIViT is a transformer framework enabling self-supervised pretraining and segmentation of 3D MR images with variable contrasts, improving adaptability and performance on heterogeneous real-world data.

Contribution

The paper introduces VIViT, a novel transformer-based framework that handles variable input contrasts in MR images for pretraining and segmentation tasks.

Findings

Outperforms CNN and ViT models in brain infarct segmentation.

Achieves high Dice scores of 0.624 and 0.883 on different tasks.

Enhances model adaptability to heterogeneous MR data.

Abstract

Self-supervised pretrain techniques have been widely used to improve the downstream tasks' performance. However, real-world magnetic resonance (MR) studies usually consist of different sets of contrasts due to different acquisition protocols, which poses challenges for the current deep learning methods on large-scale pretrain and different downstream tasks with different input requirements, since these methods typically require a fixed set of input modalities or, contrasts. To address this challenge, we propose variable-input ViT (VIViT), a transformer-based framework designed for self-supervised pretraining and segmentation finetuning for variable contrasts in each study. With this ability, our approach can maximize the data availability in pretrain, and can transfer the learned knowledge from pretrain to downstream tasks despite variations in input requirements. We validate our method on brain infarct and brain tumor segmentation, where our method outperforms current CNN and ViT-based models with a mean Dice score of 0.624 and 0.883 respectively. These results highlight the efficacy of our design for better adaptability and performance on tasks with real-world heterogeneous MR data.