Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models

TL;DR

This study investigates whether training language models on Child-Directed Language improves syntax learning, finding inconsistent benefits compared to Wikipedia data across multiple languages and benchmarks.

Contribution

The paper critically evaluates previous claims about CDL's advantages, introduces the FIT-CLAMS testing methodology, and emphasizes the importance of controlling for frequency effects in syntactic evaluation.

Findings

CDL does not consistently outperform Wikipedia in syntax learning.

Benchmark shortcomings are identified and addressed with a new methodology.

Frequency control is crucial for fair syntactic ability evaluation.

Abstract

Seminal work by Huebner et al. (2021) showed that language models (LMs) trained on English Child-Directed Language (CDL) can reach similar syntactic abilities as LMs trained on much larger amounts of adult-directed written text, suggesting that CDL could provide more effective LM training material than the commonly used internet-crawled data. However, the generalizability of these results across languages, model types, and evaluation settings remains unclear. We test this by comparing models trained on CDL vs. Wikipedia across two LM objectives (masked and causal), three languages (English, French, German), and three syntactic minimal-pair benchmarks. Our results on these benchmarks show inconsistent benefits of CDL, which in most cases is outperformed by Wikipedia models. We then identify various shortcomings in previous benchmarks, and introduce a novel testing methodology, FIT-CLAMS, which uses a frequency-controlled design to enable balanced comparisons across training corpora. Through minimal pair evaluations and regression analysis we show that training on CDL does not yield stronger generalizations for acquiring syntax and highlight the importance of controlling for frequency effects when evaluating syntactic ability.