TL;DR
This paper introduces an information-theoretic measure of storage cost in sentence comprehension, which is continuous, probabilistic, and estimable from neural language models, improving understanding of working memory load during reading.
Contribution
It proposes a novel, theory-neutral, information-theoretic metric for processing storage cost, validated through linguistic analyses and prediction of reading times.
Findings
Recovers known processing asymmetries in English syntax.
Correlates with grammar-based storage costs in annotated corpora.
Predicts reading-time variance beyond traditional predictors.
Abstract
Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have largely been formalized using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, probabilistic, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeurobiology of Language and Bilingualism · Reading and Literacy Development · Action Observation and Synchronization
