Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Clasification

TL;DR

This paper introduces MFFormer, a novel transformer-based model that effectively fuses multi-dimensional neuroimaging data from rs-fMRI and T1w sMRI to improve psychiatric disorder classification accuracy.

Contribution

The study presents a multi-dimension-embedding-aware fusion transformer that leverages both temporal and spatial information from multi-modal neuroimaging data, enhancing diagnosis of schizophrenia and bipolar disorder.

Findings

MFFormer outperforms single-modality models.

The fusion transformer improves classification accuracy.

Effective multi-dimensional feature alignment enhances results.

Abstract

Deep learning approaches, together with neuroimaging techniques, play an important role in psychiatric disorders classification. Previous studies on psychiatric disorders diagnosis mainly focus on using functional connectivity matrices of resting-state functional magnetic resonance imaging (rs-fMRI) as input, which still needs to fully utilize the rich temporal information of the time series of rs-fMRI data. In this work, we proposed a multi-dimension-embedding-aware modality fusion transformer (MFFormer) for schizophrenia and bipolar disorder classification using rs-fMRI and T1 weighted structural MRI (T1w sMRI). Concretely, to fully utilize the temporal information of rs-fMRI and spatial information of sMRI, we constructed a deep learning architecture that takes as input 2D time series of rs-fMRI and 3D volumes T1w. Furthermore, to promote intra-modality attention and information fusion across different modalities, a fusion transformer module (FTM) is designed through extensive self-attention of hybrid feature maps of multi-modality. In addition, a dimension-up and dimension-down strategy is suggested to properly align feature maps of multi-dimensional from different modalities. Experimental results on our private and public OpenfMRI datasets show that our proposed MFFormer performs better than that using a single modality or multi-modality MRI on schizophrenia and bipolar disorder diagnosis.