Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment

TL;DR

This paper introduces a framework for evaluating how well models recover specific brain response dimensions, providing a more detailed assessment of model-brain alignment beyond mere prediction accuracy.

Contribution

It presents a unified method to identify and compare reproducible brain response dimensions predicted by models and human data, enhancing diagnostic evaluation.

Findings

Early-to-intermediate visual cortex responses are low-dimensional and reproducible.

Brain-to-brain comparisons reveal consistent recoverability of certain response dimensions.

Models with similar prediction accuracy can have different recovery profiles across brain response dimensions.

Abstract

Artificial vision models are often evaluated against the human visual cortex by measuring how accurately their internal representations predict brain responses. However, prediction accuracy alone does not indicate which dimensions of the target brain's response space are recovered. Here, we introduce a unified framework for evaluating both model-brain and brain-brain alignment by identifying the response dimensions recovered by prediction. Using repeated fMRI measurements, we first identify target-brain response dimensions that can be reproducibly predicted across independent trial splits. We then predict target-brain responses from either another subject's brain responses or a vision model's internal representations, and quantify how strongly each of these reproducible response dimensions is recovered. Applying this framework to a subset of the Natural Scenes Dataset, in which eight subjects viewed the same natural images during fMRI, we find that the early-to-intermediate visual-cortex responses contain a low-dimensional set of reproducible dimensions. Brain-to-brain comparisons identify which of these dimensions are consistently recoverable from other subjects' brains, providing a diagnostic human reference rather than only a scalar benchmark. In some cases, pretrained and randomly initialized models achieve similar prediction accuracy while showing distinct recovery profiles across these response dimensions. These results show that prediction accuracy alone can mask model-brain mismatches. By making explicit which reproducible brain response dimensions are recovered by prediction, our framework provides a more diagnostic evaluation of alignment between artificial vision models and the human visual cortex.