A Tri-attention Fusion Guided Multi-modal Segmentation Network

TL;DR

This paper introduces a multi-modal segmentation network with a novel tri-attention fusion mechanism that effectively captures and leverages correlations between different MR modalities to improve brain tumor segmentation accuracy.

Contribution

The paper proposes a tri-attention fusion block that combines dual-attention and correlation attention to enhance multi-modal feature integration for segmentation.

Findings

Improved segmentation accuracy on BraTS 2018 dataset.

Effective suppression of less informative features.

Enhanced correlation learning between modalities.

Abstract

In the field of multimodal segmentation, the correlation between different modalities can be considered for improving the segmentation results. Considering the correlation between different MR modalities, in this paper, we propose a multi-modality segmentation network guided by a novel tri-attention fusion. Our network includes N model-independent encoding paths with N image sources, a tri-attention fusion block, a dual-attention fusion block, and a decoding path. The model independent encoding paths can capture modality-specific features from the N modalities. Considering that not all the features extracted from the encoders are useful for segmentation, we propose to use dual attention based fusion to re-weight the features along the modality and space paths, which can suppress less informative features and emphasize the useful ones for each modality at different positions. Since there exists a strong correlation between different modalities, based on the dual attention fusion block, we propose a correlation attention module to form the tri-attention fusion block. In the correlation attention module, a correlation description block is first used to learn the correlation between modalities and then a constraint based on the correlation is used to guide the network to learn the latent correlated features which are more relevant for segmentation. Finally, the obtained fused feature representation is projected by the decoder to obtain the segmentation results. Our experiment results tested on BraTS 2018 dataset for brain tumor segmentation demonstrate the effectiveness of our proposed method.