JGAT: a joint spatio-temporal graph attention model for brain decoding

TL;DR

This paper introduces JGAT, a novel multi-modal spatio-temporal graph attention model that effectively integrates functional and structural brain connectivity data, capturing dynamic variations for improved brain decoding.

Contribution

The paper presents JGAT, a new joint spatio-temporal graph attention network that preserves dynamic information and integrates multi-modal brain imaging data for enhanced decoding.

Findings

Successfully applied to four independent datasets, including human and animal data.

Identified informative temporal segments and dynamical pathways in brain activity.

Demonstrated improved decoding performance over traditional methods.

Abstract

The decoding of brain neural networks has been an intriguing topic in neuroscience for a well-rounded understanding of different types of brain disorders and cognitive stimuli. Integrating different types of connectivity, e.g., Functional Connectivity (FC) and Structural Connectivity (SC), from multi-modal imaging techniques can take their complementary information into account and therefore have the potential to get better decoding capability. However, traditional approaches for integrating FC and SC overlook the dynamical variations, which stand a great chance to over-generalize the brain neural network. In this paper, we propose a Joint kernel Graph Attention Network (JGAT), which is a new multi-modal temporal graph attention network framework. It integrates the data from functional Magnetic Resonance Images (fMRI) and Diffusion Weighted Imaging (DWI) while preserving the dynamic information at the same time. We conduct brain-decoding tasks with our JGAT on four independent datasets: three of 7T fMRI datasets from the Human Connectome Project (HCP) and one from animal neural recordings. Furthermore, with Attention Scores (AS) and Frame Scores (FS) computed and learned from the model, we can locate several informative temporal segments and build meaningful dynamical pathways along the temporal domain for the HCP datasets. The URL to the code of JGAT model: https://github.com/BRAINML-GT/JGAT.