Baby Scale: Investigating Models Trained on Individual Children's Language Input

TL;DR

This study examines how language models trained on child-specific data from the BabyView dataset perform across various tasks, revealing insights into linguistic development and data quality effects.

Contribution

It introduces an analysis of models trained on child language input, highlighting factors influencing learning efficiency and variability across individual children's data.

Findings

Models trained on child data perform well on grammar tasks.

Semantic and world knowledge tasks show lower scaling performance.

Word likelihoods in models correlate with children's word learning.

Abstract

Modern language models (LMs) must be trained on many orders of magnitude more words of training data than human children receive before they begin to produce useful behavior. Assessing the nature and origins of this "data gap" requires benchmarking LMs on human-scale datasets to understand how linguistic knowledge emerges from children's natural training data. Using transcripts from the BabyView dataset (videos from children ages 6-36 months), we investigate (1) scaling performance at child-scale data regimes, (2) variability in model performance across datasets from different children's experiences and linguistic predictors of dataset quality, and (3) relationships between model and child language learning outcomes. LMs trained on child data show acceptable scaling for grammar tasks, but lower scaling on semantic and world knowledge tasks than models trained on synthetic data; we also observe substantial variability on data from different children. Beyond dataset size, performance is most associated with a combination of distributional and interactional linguistic features, broadly consistent with what makes high-quality input for child language development. Finally, model likelihoods for individual words correlate with children's learning of those words, suggesting that properties of child-directed input may influence both model learning and human language development. Overall, understanding what properties make language data efficient for learning can enable more powerful small-scale language models while also shedding light on human language acquisition.