Sparse Autoencoders Bridge The Deep Learning Model and The Brain

TL;DR

This paper introduces SAE-BrainMap, a framework that aligns deep learning model representations with human brain responses using sparse autoencoders, revealing how models and the brain process visual information similarly.

Contribution

The study presents a novel method to directly connect deep learning models with brain activity, enabling detailed hierarchical mapping without additional training.

Findings

Strong correlation between SAE units and fMRI signals (up to 0.76)

SAE units preserve functional structure of brain regions

Hierarchical mapping reveals information flow in visual pathway

Abstract

We present SAE-BrainMap, a novel framework that directly aligns deep learning visual model representations with voxel-level fMRI responses using sparse autoencoders (SAEs). First, we train layer-wise SAEs on model activations and compute the correlations between SAE unit activations and cortical fMRI signals elicited by the same natural image stimuli with cosine similarity, revealing strong activation correspondence (maximum similarity up to 0.76). Depending on this alignment, we construct a voxel dictionary by optimally assigning the most similar SAE feature to each voxel, demonstrating that SAE units preserve the functional structure of predefined regions of interest (ROIs) and exhibit ROI-consistent selectivity. Finally, we establish fine-grained hierarchical mapping between model layers and the human ventral visual pathway, also by projecting voxel dictionary activations onto individual cortical surfaces, we visualize the dynamic transformation of the visual information in deep learning models. It is found that ViT-B/16$_{CLIP}$ tends to utilize low-level information to generate high-level semantic information in the early layers and reconstructs the low-dimension information later. Our results establish a direct, downstream-task-free bridge between deep neural networks and human visual cortex, offering new insights into model interpretability.