Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

TL;DR

This study compares neural language models with psychologically plausible models across multi-modal, bilingual datasets, revealing that psychologically plausible models, especially embodied ones, better predict brain activity in diverse contexts.

Contribution

It provides a comprehensive comparison showing psychologically plausible models outperform neural models in predicting brain activity across multiple modalities and languages.

Findings

Psychologically plausible models outperform neural models in encoding brain activity.

Embodied models show superior performance at word and discourse levels.

Models predict brain activation across various regions in English and Chinese.

Abstract

Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.