Collateral facilitation in humans and language models

TL;DR

This study compares human and language model predictions during language comprehension, revealing that both exhibit similar facilitation effects for anomalous words related to context, which informs our understanding of language processing.

Contribution

The paper demonstrates that contemporary transformer language models show similar collateral facilitation effects as humans when processing anomalous words in context.

Findings

Language models exhibit facilitation for anomalous words related to context.

Humans and models show similar prediction patterns during language comprehension.

Implications for understanding human and AI language processing.

Abstract

Are the predictions of humans and language models affected by similar things? Research suggests that while comprehending language, humans make predictions about upcoming words, with more predictable words being processed more easily. However, evidence also shows that humans display a similar processing advantage for highly anomalous words when these words are semantically related to the preceding context or to the most probable continuation. Using stimuli from 3 psycholinguistic experiments, we find that this is also almost always also the case for 8 contemporary transformer language models (BERT, ALBERT, RoBERTa, XLM-R, GPT-2, GPT-Neo, GPT-J, and XGLM). We then discuss the implications of this phenomenon for our understanding of both human language comprehension and the predictions made by language models.