Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models

TL;DR

This study demonstrates that multimodal models with attention mechanisms better align with human predictions of upcoming words, especially when visual cues are integrated, highlighting the importance of multimodality and attention in natural language understanding.

Contribution

It shows that multimodal models with attention mechanisms more closely replicate human language prediction, emphasizing the role of visual cues and attention in alignment.

Findings

Multimodal models align more closely with human predictions than unimodal models.

Including attention mechanisms doubles the alignment with human judgments.

Visual cues significantly improve model-human alignment in language prediction.

Abstract

The potential of multimodal generative artificial intelligence (mAI) to replicate human grounded language understanding, including the pragmatic, context-rich aspects of communication, remains to be clarified. Humans are known to use salient multimodal features, such as visual cues, to facilitate the processing of upcoming words. Correspondingly, multimodal computational models can integrate visual and linguistic data using a visual attention mechanism to assign next-word probabilities. To test whether these processes align, we tasked both human participants (N = 200) as well as several state-of-the-art computational models with evaluating the predictability of forthcoming words after viewing short audio-only or audio-visual clips with speech. During the task, the model's attention weights were recorded and human attention was indexed via eye tracking. Results show that predictability estimates from humans aligned more closely with scores generated from multimodal models vs. their unimodal counterparts. Furthermore, including an attention mechanism doubled alignment with human judgments when visual and linguistic context facilitated predictions. In these cases, the model's attention patches and human eye tracking significantly overlapped. Our results indicate that improved modeling of naturalistic language processing in mAI does not merely depend on training diet but can be driven by multimodality in combination with attention-based architectures. Humans and computational models alike can leverage the predictive constraints of multimodal information by attending to relevant features in the input.