Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe

TL;DR

This paper introduces a comprehensive brain encoding model trained on diverse neuroimaging data, leveraging retinotopy for inductive bias and a novel All-for-One training approach to handle heterogeneity across individuals and modalities.

Contribution

The paper presents the most extensive brain encoding model to date, utilizing a new training recipe and biological insights to improve brain response prediction and decoding.

Findings

Pre-trained model outperforms standard vision backbones.

Model effectively predicts voxel responses across diverse datasets.

Demonstrates successful brain decoding applications.

Abstract

Brain encoding models aim to predict brain voxel-wise responses to stimuli images, replicating brain signals captured by neuroimaging techniques. There is a large volume of publicly available data, but training a comprehensive brain encoding model is challenging. The main difficulties stem from a) diversity within individual brain, with functional heterogeneous brain regions; b) diversity of brains from different subjects, due to genetic and developmental differences; c) diversity of imaging modalities and processing pipelines. We use this diversity to our advantage by introducing the All-for-One training recipe, which divides the challenging one-big-model problem into multiple small models, with the small models aggregating the knowledge while preserving the distinction between the different functional regions. Agnostic of the training recipe, we use biological knowledge of the brain, specifically retinotopy, to introduce inductive bias to learn a 3D brain-to-image mapping that ensures a) each neuron knows which image regions and semantic levels to gather information, and b) no neurons are left behind in the model. We pre-trained a brain encoding model using over one million data points from five public datasets spanning three imaging modalities. To the best of our knowledge, this is the most comprehensive brain encoding model to the date. We demonstrate the effectiveness of the pre-trained model as a drop-in replacement for commonly used vision backbone models. Furthermore, we demonstrate the application of the model to brain decoding. Code and the model checkpoint will be made available.