Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

TL;DR

This study systematically compares multilingual language models trained on child-directed speech and multi-domain data in English and French, revealing how training data type influences linguistic task performance.

Contribution

It extends BabyBERTa to bilingual English-French scenarios, introduces new multilingual resources, and analyzes the effects of different training corpora on language understanding.

Findings

Wikipedia training benefits semantic tasks

Child-directed speech improves grammatical judgments in monolingual settings

Bilingual pretraining enhances textual entailment, especially for French

Abstract

Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.