fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding

TL;DR

fMRI-LM is a novel foundational model that links fMRI brain activity with language, enabling semantic understanding and cross-modal reasoning through a three-stage training process.

Contribution

The paper introduces a three-stage framework for creating a language-aligned fMRI model, including neural tokenization, joint modeling with LLMs, and instruction tuning.

Findings

Achieves strong zero-shot and few-shot performance on benchmarks.

Effectively models fMRI signals with language representations.

Supports diverse downstream applications with high-level semantic understanding.

Abstract

Recent advances in multimodal large language models (LLMs) have enabled unified reasoning across images, audio, and video, but extending such capability to brain imaging remains largely unexplored. Bridging this gap is essential to link neural activity with semantic cognition and to develop cross-modal brain representations. To this end, we present fMRI-LM, a foundational model that bridges functional MRI (fMRI) and language through a three-stage framework. In Stage 1, we learn a neural tokenizer that maps fMRI into discrete tokens embedded in a language-consistent space. In Stage 2, a pretrained LLM is adapted to jointly model fMRI tokens and text, treating brain activity as a sequence that can be temporally predicted and linguistically described. To overcome the lack of natural fMRI-text pairs, we construct a large descriptive corpus that translates diverse imaging-based features into structured textual descriptors, capturing the low-level organization of fMRI signals. In Stage 3, we perform multi-task, multi-paradigm instruction tuning to endow fMRI-LM with high-level semantic understanding, supporting diverse downstream applications. Across various benchmarks, fMRI-LM achieves strong zero-shot and few-shot performance, and adapts efficiently with parameter-efficient tuning (LoRA), establishing a scalable pathway toward a language-aligned, universal model for structural and semantic understanding of fMRI.