Deconstructing sentence disambiguation by joint latent modeling of reading paradigms: LLM surprisal is not enough

TL;DR

This paper introduces a latent-process mixture model that better captures human reading behavior during sentence disambiguation tasks, outperforming GPT-2 surprisal-based models in predicting reading patterns.

Contribution

The study presents a novel mixture model that distinguishes different processing costs and accounts for inattentive reading, improving prediction of human reading behavior over existing surprisal-based models.

Findings

The model accurately reproduces rereading and comprehension patterns.

It outperforms GPT-2 surprisal models in predictive accuracy.

It provides more realistic estimates of processing costs.

Abstract

Using temporarily ambiguous garden-path sentences ("While the team trained the striker wondered ...") as a test case, we present a latent-process mixture model of human reading behavior across four different reading paradigms (eye tracking, uni- and bidirectional self-paced reading, Maze). The model distinguishes between garden-path probability, garden-path cost, and reanalysis cost, and yields more realistic processing cost estimates by taking into account trials with inattentive reading. We show that the model is able to reproduce empirical patterns with regard to rereading behavior, comprehension question responses, and grammaticality judgments. Cross-validation reveals that the mixture model also has better predictive fit to human reading patterns and end-of-trial task data than a mixture-free model based on GPT-2-derived surprisal values. We discuss implications for future work.