Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding

TL;DR

This paper introduces STDA-SwiFT, a transformer-based model that leverages large-scale fMRI data to learn transferable brain representations, significantly improving the decoding of task-evoked neural activity across various cognitive domains.

Contribution

The study presents a novel transformer model with spatial-temporal divided attention and contrastive learning, enabling transfer learning from large fMRI datasets to enhance decoding accuracy.

Findings

Improved decoding performance across sensory and cognitive tasks.

Larger receptor fields enhance decoding with memory-efficient attention.

Pretraining on large datasets benefits small-sample fine-tuning.

Abstract

A fundamental challenge in neuroscience is to decode mental states from brain activity. While functional magnetic resonance imaging (fMRI) offers a non-invasive approach to capture brain-wide neural dynamics with high spatial precision, decoding from fMRI data -- particularly from task-evoked activity -- remains challenging due to its high dimensionality, low signal-to-noise ratio, and limited within-subject data. Here, we leverage recent advances in computer vision and propose STDA-SwiFT, a transformer-based model that learns transferable representations from large-scale fMRI datasets via spatial-temporal divided attention and self-supervised contrastive learning. Using pretrained voxel-wise representations from 995 subjects in the Human Connectome Project (HCP), we show that our model substantially improves downstream decoding performance of task-evoked activity across multiple sensory and cognitive domains, even with minimal data preprocessing. We demonstrate performance gains from larger receptor fields afforded by our memory-efficient attention mechanism, as well as the impact of functional relevance in pretraining data when fine-tuning on small samples. Our work showcases transfer learning as a viable approach to harness large-scale datasets to overcome challenges in decoding brain activity from fMRI data.