Latent Representation Learning for Multimodal Brain Activity Translation

TL;DR

This paper introduces SAMBA, a framework that learns a unified latent space to integrate diverse neuroimaging data, enabling better understanding and classification of brain activity across modalities.

Contribution

SAMBA combines novel spectral filtering, graph attention, and recurrent layers to align and learn rich representations from multimodal brain data.

Findings

Successfully classifies external stimuli from brain activity

Learns modality-independent brain representations

Bridges spatial and temporal resolution gaps in neuroimaging

Abstract

Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts.