Language acquisition: do children and language models follow similar learning stages?

TL;DR

This study compares the language learning stages of children and GPT-2 models, revealing similarities in systematic learning order and parallel development, but also highlighting key differences in their acquisition processes.

Contribution

The paper provides the first detailed comparison of language acquisition stages between children and deep language models, revealing both parallels and divergences.

Findings

Language models learn linguistic skills in a systematic order similar to children.

Learning tasks that are learned last improve from early training stages.

Some learning stages are shared between children and models, others are not.

Abstract

During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master increasingly complex syntactic structures. However, the computational principles that lead to this learning trajectory remain largely unknown. To investigate this, we here compare the learning trajectories of deep language models to those of children. Specifically, we test whether, during its training, GPT-2 exhibits stages of language acquisition comparable to those observed in children aged between 18 months and 6 years. For this, we train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step, using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. We then compare these evaluations with the behavior of 54 children during language production. Our analyses reveal three main findings. First, similarly to children, the language models tend to learn linguistic skills in a systematic order. Second, this learning scheme is parallel: the language tasks that are learned last improve from the very first training steps. Third, some - but not all - learning stages are shared between children and these language models. Overall, these results shed new light on the principles of language acquisition, and highlight important divergences in how humans and modern algorithms learn to process natural language.