Evaluating Neural Language Models as Cognitive Models of Language Acquisition

TL;DR

This paper critically assesses neural language models as models of language acquisition, highlighting limitations of current benchmarks and proposing more linguistically and psychologically relevant evaluation methods.

Contribution

It identifies shortcomings of existing syntactic benchmarks and advocates for using curated datasets aligned with linguistic theory and child language acquisition research.

Findings

Current benchmarks lack structural diversity of natural language.

Small-scale training data can be matched by simple baseline models.

Neural models evaluate sentences inconsistently with human judgments.

Abstract

The success of neural language models (LMs) on many technological tasks has brought about their potential relevance as scientific theories of language despite some clear differences between LM training and child language acquisition. In this paper we argue that some of the most prominent benchmarks for evaluating the syntactic capacities of LMs may not be sufficiently rigorous. In particular, we show that the template-based benchmarks lack the structural diversity commonly found in the theoretical and psychological studies of language. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We advocate for the use of the readily available, carefully curated datasets that have been evaluated for gradient acceptability by large pools of native speakers and are designed to probe the structural basis of grammar specifically. On one such dataset, the LI-Adger dataset, LMs evaluate sentences in a way inconsistent with human language users. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.