In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain

TL;DR

This paper introduces a novel in silico, data-driven method using transformer models to map and understand complex visual category selectivity across the entire human brain, surpassing traditional linear approaches.

Contribution

It presents a transformer-based encoder-decoder model with cross-attention for nonlinear brain-to-image feature mapping and a new method to synthesize images that activate specific brain regions.

Findings

Revealed complex, compositional selectivity in brain regions.

Validated in silico predictions across multiple subjects.

Provided a framework for hypothesis generation in visual neuroscience.

Abstract

A fine-grained account of functional selectivity in the cortex is essential for understanding how visual information is processed and represented in the brain. Classical studies using designed experiments have identified multiple category-selective regions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven discovery methods have sought to address this limitation but are often limited by simple, typically linear encoding models. We propose an in silico approach for data-driven discovery of novel category-selectivity hypotheses based on an encoder-decoder transformer model. The architecture incorporates a brain-region to image-feature cross-attention mechanism, enabling nonlinear mappings between high-dimensional deep network features and semantic patterns encoded in the brain activity. We further introduce a method to characterize the selectivity of individual parcels by leveraging diffusion-based image generative models and large-scale datasets to synthesize and select images that maximally activate each parcel. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts, which we validate in silico both within and across subjects. Using a brain encoder as a "digital twin" offers a powerful, data-driven framework for generating and testing hypotheses about visual selectivity in the human brain - hypotheses that can guide future fMRI experiments. Our code is available at: https://kriegeskorte-lab.github.io/in-silico-mapping/ .