Successive-cyclic movement in humans and neural language models: testing wh-filler-gap dependencies

TL;DR

This study shows that language models like GPT-2 do not process complex sentence structures in the same way humans do, suggesting they lack understanding of abstract grammar.

Contribution

The study provides empirical evidence that GPT-style models fail to capture human-like syntactic processing of wh-filler-gap dependencies.

Findings

Language models failed to replicate human processing facilitation for canonical and elided dependencies.

One model showed an inverse effect, relying on surface cues instead of abstract syntax.

The findings support the Poverty of the Stimulus argument against models' syntactic competence.

Abstract

This study investigates whether auto-regressive language models (GPT-2, GPT-Neo, OPT) replicate human-like sensitivity to covert intermediate phrasal structures (CP vs. NP) during the processing of wh-filler-gap dependencies. We extend this inquiry to backward sluicing, an elliptical construction that provides a robust test for the representation of abstract syntactic structure. Across two experiments measuring processing difficulty via surprisal, we found a significant divergence from established human processing patterns. We found that the models failed to reproduce the human processing facilitation for both canonical and elided dependencies. One model, in fact, showed an inverse effect, a pattern suggesting a reliance on surface-level cues rather than abstract hierarchical representations. We take these findings as evidence that the tested GPT-style models are insufficient for deriving knowledge of covert syntactic structures. This failure lends empirical support to the Poverty of the Stimulus (PoS) argument, and also highlights a significant gap in the cognitive plausibility of contemporary NLMs as models of human syntactic competence.

Figures