A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Models

TL;DR

This paper evaluates whether neural language models can learn complex syntax with limited data, testing the Poverty of the Stimulus hypothesis by introducing a benchmark and analyzing model generalization and biases.

Contribution

It introduces extposhbench, a benchmark for assessing syntactic generalization, and evaluates neural models' capabilities and limitations in relation to innate linguistic constraints.

Findings

Neural models show some generalization without explicit evidence.

Models are less data-efficient than children in learning syntax.

Cognitive biases improve syntactic competence but not all aspects of generalization.

Abstract

How can children acquire native-level syntax from limited input? According to the Poverty of the Stimulus Hypothesis (PoSH), the linguistic input children receive is insufficient to explain certain generalizations that are robustly learned; innate linguistic constraints, many have argued, are thus necessary to explain language learning. Neural language models, which lack such language-specific constraints in their design, offer a computational test of this longstanding (but controversial) claim. We introduce \poshbench, a training-and-evaluation suite targeting question formation, islands to movement, and other English phenomena at the center of the PoSH arguments. Training Transformer models on 10--50M words of developmentally plausible text, we find indications of generalization on all phenomena even without direct positive evidence -- yet neural models remain less data-efficient and their generalizations are weaker than those of children. We further enhance our models with three recently proposed cognitively motivated inductive biases. We find these biases improve general syntactic competence but not \poshbench performance. Our findings challenge the claim that innate syntax is the only possible route to generalization, while suggesting that human-like data efficiency requires inductive biases beyond those tested here.