Disentangling the Factors of Convergence between Brains and Computer Vision Models

TL;DR

This study systematically investigates how model size, training data, and architecture influence the development of brain-like visual representations in self-supervised vision transformers, revealing a developmental trajectory aligned with human cortical features.

Contribution

It provides a comprehensive analysis of factors affecting brain-model similarity, highlighting the emergence of human-like representations during training and their relation to cortical development.

Findings

Larger models trained on human-centric images show higher brain similarity.

Brain-like representations emerge progressively, first aligning with sensory cortices, then with higher-order areas.

Representation development correlates with cortical features like expansion, thickness, and timescales.

Abstract

Many AI models trained on natural images develop representations that resemble those of the human brain. However, the factors that drive this brain-model similarity remain poorly understood. To disentangle how the model, training and data independently lead a neural network to develop brain-like representations, we trained a family of self-supervised vision transformers (DINOv3) that systematically varied these different factors. We compare their representations of images to those of the human brain recorded with both fMRI and MEG, providing high resolution in spatial and temporal analyses. We assess the brain-model similarity with three complementary metrics focusing on overall representational similarity, topographical organization, and temporal dynamics. We show that all three factors - model size, training amount, and image type - independently and interactively impact each of these brain similarity metrics. In particular, the largest DINOv3 models trained with the most human-centric images reach the highest brain-similarity. This emergence of brain-like representations in AI models follows a specific chronology during training: models first align with the early representations of the sensory cortices, and only align with the late and prefrontal representations of the brain with considerably more training. Finally, this developmental trajectory is indexed by both structural and functional properties of the human cortex: the representations that are acquired last by the models specifically align with the cortical areas with the largest developmental expansion, thickness, least myelination, and slowest timescales. Overall, these findings disentangle the interplay between architecture and experience in shaping how artificial neural networks come to see the world as humans do, thus offering a promising framework to understand how the human brain comes to represent its visual world.