On the Proper Treatment of Units in Surprisal Theory

TL;DR

This paper clarifies how to properly handle units in surprisal theory, emphasizing the importance of explicit choices in tokenization and analysis regions for linguistic predictability models.

Contribution

It introduces a unified framework for analyzing surprisal over arbitrary units, disentangling unit definition from evaluation regions.

Findings

Surprisal analysis depends on explicit unit choices rather than fixed tokenization.

Tokenization should be considered an implementation detail, not a core scientific primitive.

The framework allows for more accurate and flexible linguistic predictability modeling.

Abstract

Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.