Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

TL;DR

This paper evaluates how well transformer-based language models mimic human sentence processing, revealing their limitations in capturing complex agreement attraction effects and emphasizing the need for comprehensive testing.

Contribution

It provides a systematic, comparative analysis of multiple transformer models on agreement attraction, highlighting their shortcomings in modeling human-like morphosyntactic processing.

Findings

Transformers align with human data on simple configurations

Performance drops on complex relative clause structures

Models fail to replicate human asymmetric interference patterns

Abstract

Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.