3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

TL;DR

This paper introduces a novel 3D Transformer-based method with a fusion-head self-attention mechanism for brain tumor segmentation in MRI images, addressing limitations of previous 2D and convolutional approaches.

Contribution

The work proposes a new 3D Transformer model with a fusion-head self-attention mechanism and a plug-and-play deformable feature extractor for improved brain tumor segmentation.

Findings

Achieved superior segmentation performance on BRATS datasets.

Outperformed several state-of-the-art methods.

Demonstrated effective modeling of long-range spatial dependencies.

Abstract

Magnetic resonance imaging (MRI) is critically important for brain mapping in both scientific research and clinical studies. Precise segmentation of brain tumors facilitates clinical diagnosis, evaluations, and surgical planning. Deep learning has recently emerged to improve brain tumor segmentation and achieved impressive results. Convolutional architectures are widely used to implement those neural networks. By the nature of limited receptive fields, however, those architectures are subject to representing long-range spatial dependencies of the voxel intensities in MRI images. Transformers have been leveraged recently to address the above limitations of convolutional networks. Unfortunately, the majority of current Transformers-based methods in segmentation are performed with 2D MRI slices, instead of 3D volumes. Moreover, it is difficult to incorporate the structures between layers because each head is calculated independently in the Multi-Head Self-Attention mechanism (MHSA). In this work, we proposed a 3D Transformer-based segmentation approach. We developed a Fusion-Head Self-Attention mechanism (FHSA) to combine each attention head through attention logic and weight mapping, for the exploration of the long-range spatial dependencies in 3D MRI images. We implemented a plug-and-play self-attention module, named the Infinite Deformable Fusion Transformer Module (IDFTM), to extract features on any deformable feature maps. We applied our approach to the task of brain tumor segmentation, and assessed it on the public BRATS datasets. The experimental results demonstrated that our proposed approach achieved superior performance, in comparison to several state-of-the-art segmentation methods.