Are words equally surprising in audio and audio-visual comprehension?

TL;DR

This study investigates how visual cues influence spoken language comprehension by comparing neural responses in audio-only and audio-visual settings, revealing differences in cognitive effort and the effectiveness of language models.

Contribution

It provides new insights into how multimodal information affects language processing and evaluates the predictive power of different language models for neural responses.

Findings

Transformer models better predict N400 in audio-only conditions.

2-gram models are more effective in multimodal settings.

Cognitive effort varies significantly between audio-only and audio-visual conditions.

Abstract

We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on spoken language comprehension. We compare the ERP signature (N400) associated with each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (which quantify the predictability of words in their lexical context) are generated on the basis of different types of language models (specifically n-gram and Transformer models) that predict N400 responses for each word. Our results indicate that cognitive effort differs significantly between multimodal and unimodal settings. In addition, our findings suggest that while Transformer-based models, which have access to a larger lexical context, provide a better fit in the audio-only setting, 2-gram language models are more effective in the multimodal setting. This highlights the significant impact of local lexical context on cognitive processing in a multimodal environment.