Information-Theoretic Storage Cost in Sentence Comprehension

TL;DR

This paper introduces an information-theoretic measure of storage cost in sentence comprehension, which is continuous, probabilistic, and estimable from neural language models, improving understanding of working memory load during reading.

Contribution

It proposes a novel, theory-neutral, information-theoretic metric for processing storage cost, validated through linguistic analyses and prediction of reading times.

Findings

Recovers known processing asymmetries in English syntax.

Correlates with grammar-based storage costs in annotated corpora.

Predicts reading-time variance beyond traditional predictors.

Abstract

Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have largely been formalized using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, probabilistic, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storage cost in a syntactically-annotated corpus, and (iii) predicts reading-time variance in two large-scale naturalistic datasets over and above baseline models with traditional information-based predictors. Our code is available at https://github.com/kohei-kaji/info-storage.