Medical Transformer: Universal Brain Encoder for 3D MRI Analysis

TL;DR

This paper introduces Medical Transformer, a parameter-efficient transfer learning framework that models 3D MRI data as sequences of 2D slices, leveraging multi-view information and self-supervised pre-training to improve performance on various brain MRI tasks.

Contribution

The paper presents a novel multi-view Transformer architecture for 3D MRI analysis, pre-trained with self-supervised learning, achieving superior results with fewer parameters.

Findings

Outperforms state-of-the-art transfer learning methods.

Reduces model parameters by up to 92%.

Effective across multiple brain MRI tasks.

Abstract

Transfer learning has gained attention in medical image analysis due to limited annotated 3D medical datasets for training data-driven deep learning models in the real world. Existing 3D-based methods have transferred the pre-trained models to downstream tasks, which achieved promising results with only a small number of training samples. However, they demand a massive amount of parameters to train the model for 3D medical imaging. In this work, we propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of 2D image slices. To make a high-level representation in 3D-form empowering spatial relations better, we take a multi-view approach that leverages plenty of information from the three planes of 3D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pre-train the model in a self-supervised learning manner for masked encoding vector prediction as a proxy task, using a large-scale normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pre-trained model is evaluated on three downstream tasks: (i) brain disease diagnosis, (ii) brain age prediction, and (iii) brain tumor segmentation, which are actively studied in brain MRI research. The experimental results show that our Medical Transformer outperforms the state-of-the-art transfer learning methods, efficiently reducing the number of parameters up to about 92% for classification and