Vision-Language Models Align with Human Neural Representations in Concept Processing

TL;DR

This study evaluates how different vision-language models align with human brain responses during concept processing, revealing that some models better mimic human-like understanding and are more brain-aligned than others.

Contribution

It systematically compares various VLM architectures and their alignment with neural data, highlighting the impact of training and model design on brain similarity.

Findings

VLMs outperform language-only models in brain alignment.

Some models like LXMERT and IDEFICS2 learn more human-like concepts.

Vision-language encoders are more brain-aligned than generative VLMs.

Abstract

Recent studies suggest that transformer-based vision-language models (VLMs) capture the multimodality of concept processing in the human brain. However, a systematic evaluation exploring different types of VLM architectures and the role played by visual and textual context is still lacking. Here, we analyse multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. We measure the alignment between concept representations by models and existing (fMRI) brain responses to concept words presented in two experimental conditions, where either visual (pictures) or textual (sentences) context is provided. Our results reveal that VLMs outperform the language-only counterparts in both experimental conditions. However, controlled ablation studies show that only for some VLMs, such as LXMERT and IDEFICS2, brain alignment stems from genuinely learning more human-like concepts during pretraining, while others are highly sensitive to the context provided at inference. Additionally, we find that vision-language encoders are more brain-aligned than more recent, generative VLMs. Altogether, our study shows that VLMs align with human neural representations in concept processing, while highlighting differences among architectures. We open-source code and materials to reproduce our experiments at: https://github.com/dmg-illc/vl-concept-processing.