Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

TL;DR

Brain-OF is a unified foundation model that integrates fMRI, EEG, and MEG data, enabling comprehensive neural signal analysis through novel resolution and semantic alignment techniques.

Contribution

It introduces a multimodal brain foundation model with a shared semantic space, dual-domain pretraining, and specialized attention mechanisms for diverse neuroimaging modalities.

Findings

Pretrained on 40 datasets, surpasses existing models in various tasks.

Effectively handles heterogeneous resolutions and semantic shifts.

Demonstrates the advantages of joint multimodal and dual-domain learning.

Abstract

Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across different neuroimaging techniques. This limitation largely arises from severe semantic heterogeneity and resolution discrepancies among modalities. To address these challenges, we propose Brain-OF, an omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space. To further manage semantic shifts, the Brain-OF backbone integrates DINT attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, to explicitly internalize the characteristics of neural activity through self-supervised learning, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs brain signals in both the time and frequency domains. Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint multimodal integration and dual-domain pretraining.