Language models align with brain regions that represent concepts across modalities

TL;DR

This study shows that language models align with brain regions involved in conceptual meaning across modalities, indicating they may internally represent cross-modal concepts, as evidenced by better predictions in meaning-consistent brain areas.

Contribution

It introduces a novel measure of meaning consistency across modalities and demonstrates that language models align with brain regions representing cross-modal concepts.

Findings

Language models predict brain signals better in meaning-consistent areas.

Models align with regions involved in conceptual, not just linguistic, processing.

Cross-modal meaning influences model-brain alignment.

Abstract

Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.