From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure

TL;DR

This paper introduces a surface-based brain decoding method that models cortical surface data and individual anatomy to improve visual stimulus reconstruction from fMRI signals, achieving better accuracy and interpretability.

Contribution

It presents a novel sphere tokenizer, integrates structural MRI data for personalized encoding, and employs a mixup strategy, advancing brain decoding techniques beyond existing flattening approaches.

Findings

Outperforms state-of-the-art reconstruction methods.

Enhances biological interpretability of neural decoding.

Improves generalization across different individuals.

Abstract

Reconstructing visual stimuli from human brain activity (e.g., fMRI) bridges neuroscience and computer vision by decoding neural representations. However, existing methods often overlook critical brain structure-function relationships, flattening spatial information and neglecting individual anatomical variations. To address these issues, we propose (1) a novel sphere tokenizer that explicitly models fMRI signals as spatially coherent 2D spherical data on the cortical surface; (2) integration of structural MRI (sMRI) data, enabling personalized encoding of individual anatomical variations; and (3) a positive-sample mixup strategy for efficiently leveraging multiple fMRI scans associated with the same visual stimulus. Collectively, these innovations enhance reconstruction accuracy, biological interpretability, and generalizability across individuals. Experiments demonstrate superior reconstruction performance compared to SOTA methods, highlighting the effectiveness and interpretability of our biologically informed approach.