The Grammar-Learning Trajectories of Neural Language Models

TL;DR

This paper investigates the learning trajectories of neural language models, revealing consistent stages of linguistic acquisition across different models and highlighting an underlying inductive bias in their learning process.

Contribution

It demonstrates that diverse neural language models follow similar learning trajectories, providing insights into their implicit linguistic representations and developmental stages.

Findings

Models acquire linguistic phenomena in similar order despite differences.

Learning trajectories are approximately one-dimensional and predictable.

Phenomena clusters, especially morphological, progress together.

Abstract

The learning trajectories of linguistic phenomena in humans provide insight into linguistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker. To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different models are similar enough in the generalizations they make. In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena. Taking inspiration from psycholinguistics, we argue that studying this inductive bias is an opportunity to study the linguistic representation implicit in NLMs. Leveraging these findings, we compare the relative performance on different phenomena at varying learning stages with simpler reference models. Results suggest that NLMs exhibit consistent "developmental" stages. Moreover, we find the learning trajectory to be approximately one-dimensional: given an NLM with a certain overall performance, it is possible to predict what linguistic generalizations it has already acquired. Initial analysis of these stages presents phenomena clusters (notably morphological ones), whose performance progresses in unison, suggesting a potential link between the generalizations behind them.