So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements

TL;DR

This study compares human and computational language model predictions of N400 amplitude, finding that models like GPT-3, RoBERTa, and ALBERT better predict neural responses than human predictability judgments, highlighting the influence of surface-level language statistics.

Contribution

It demonstrates that computational models' predictions align more closely with neural responses than human judgments, challenging assumptions about human predictive processing in language comprehension.

Findings

Language models better predict N400 amplitude than humans.

Surface-level statistical information influences neural responses more than human predictability.

Computational models' predictions correlate strongly with neural data.

Abstract

More predictable words are easier to process - they are read faster and elicit smaller neural signals associated with processing difficulty, most notably, the N400 component of the event-related brain potential. Thus, it has been argued that prediction of upcoming words is a key component of language comprehension, and that studying the amplitude of the N400 is a valuable way to investigate the predictions we make. In this study, we investigate whether the linguistic predictions of computational language models or humans better reflect the way in which natural language stimuli modulate the amplitude of the N400. One important difference in the linguistic predictions of humans versus computational language models is that while language models base their predictions exclusively on the preceding linguistic context, humans may rely on other factors. We find that the predictions of three top-of-the-line contemporary language models - GPT-3, RoBERTa, and ALBERT - match the N400 more closely than human predictions. This suggests that the predictive processes underlying the N400 may be more sensitive to the surface-level statistics of language than previously thought.