Unified Cross-Modal Attention-Mixer Based Structural-Functional Connectomics Fusion for Neuropsychiatric Disorder Diagnosis

TL;DR

This paper introduces ConneX, a novel multimodal fusion framework combining cross-attention and MLP-Mixer to improve neuropsychiatric disorder diagnosis using structural and functional connectomics data.

Contribution

It presents a new fusion method that effectively integrates multimodal brain data with cross-attention and MLP-Mixer, enhancing diagnostic accuracy.

Findings

Improved diagnostic performance on clinical datasets.

Effective capture of intra- and inter-modal interactions.

Robustness demonstrated across multiple datasets.

Abstract

Gaining insights into the structural and functional mechanisms of the brain has been a longstanding focus in neuroscience research, particularly in the context of understanding and treating neuropsychiatric disorders such as Schizophrenia (SZ). Nevertheless, most of the traditional multimodal deep learning approaches fail to fully leverage the complementary characteristics of structural and functional connectomics data to enhance diagnostic performance. To address this issue, we proposed ConneX, a multimodal fusion method that integrates cross-attention mechanism and multilayer perceptron (MLP)-Mixer for refined feature fusion. Modality-specific backbone graph neural networks (GNNs) were firstly employed to obtain feature representation for each modality. A unified cross-modal attention network was then introduced to fuse these embeddings by capturing intra- and inter-modal interactions, while MLP-Mixer layers refined global and local features, leveraging higher-order dependencies for end-to-end classification with a multi-head joint loss. Extensive evaluations demonstrated improved performance on two distinct clinical datasets, highlighting the robustness of our proposed framework.