BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity

TL;DR

BaRISTA introduces a flexible spatiotemporal transformer model for intracranial neural data, demonstrating that multi-scale spatial encoding improves decoding and neural reconstruction, advancing understanding of brain network patterns.

Contribution

The paper presents a novel spatiotemporal transformer with adjustable spatial encoding scales and a self-supervised task, enabling better modeling of multiregional brain activity.

Findings

Larger spatial scales improve downstream decoding performance.

Region-level token encoding maintains neural reconstruction accuracy.

Adjusting spatial encoding scales significantly impacts model performance.

Abstract

Intracranial recordings have opened a unique opportunity to simultaneously measure activity across multiregional networks in the human brain. Recent works have focused on developing transformer-based neurofoundation models of such recordings that can generalize across subjects and datasets. However, these recordings exhibit highly complex spatiotemporal interactions across diverse spatial scales, from the single-channel scale to the scale of brain regions. As such, there remain critical open questions regarding how best to encode spatial information and how to design self-supervision tasks that enable the learning of brain network patterns and enhance downstream decoding performance using such high-dimensional, multiregional recordings. To allow for exploring these questions, we propose a new spatiotemporal transformer model of multiregional neural activity and a corresponding self-supervised masked latent reconstruction task, designed to enable flexibility in the spatial scale used for token encoding and masking. Applying this model on publicly available multiregional intracranial electrophysiology (iEEG) data, we demonstrate that adjusting the spatial scale for both token encoding and masked reconstruction significantly impacts downstream decoding. Further, we find that spatial encoding at larger scales than channel-level encoding, which is commonly used in existing iEEG transformer models, improves downstream decoding performance. Finally, we demonstrate that our method allows for region-level token encoding while also maintaining accurate channel-level neural reconstruction. Taken together, our modeling framework enables exploration of the spatial scales used for token encoding and masking, reveals their importance towards self-supervised pretraining of neurofoundation models of multiregional human brain activity, and enhances downstream decoding performance.