Generalized Measures of Anticipation and Responsivity in Online Language Processing

TL;DR

This paper introduces a generalized framework for measuring anticipation and responsivity in online language processing, improving predictive power over traditional entropy and surprisal measures, and demonstrating empirical benefits in predicting human reading behaviors.

Contribution

It provides a formal definition of new anticipatory and responsive measures, enabling more expressive analysis of language processing beyond standard metrics.

Findings

Enhanced predictive power for human cloze completion probabilities

Better prediction of neural responses like N400 amplitudes

Improved correlation with reading time data

Abstract

We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.