What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance

TL;DR

This study investigates how different dataset compositions affect small language model performance in sample-efficient training, revealing that dataset choice should depend on model size for optimal results.

Contribution

It provides empirical insights into the impact of dataset source and complexity on small language model training efficiency and performance.

Findings

Gutenberg dataset improves performance for larger models.

Child-directed speech and synthetic data underperform across sizes.

Optimal dataset depends on model size and capacity.

Abstract

We explore the impact of pre-training data composition on the performance of small language models in a sample-efficient setting. Using datasets limited to 10 million words, we evaluate several dataset sources, including child-directed speech (CHILDES), classic books (Gutenberg), synthetic data (TinyStories), and a mix of these (Mix) across different model sizes ranging from 18 million to 705 million parameters. Our experiments show that smaller models (e.g., GPT2-97M, GPT2-705M, Llama-360M) perform better when trained on more complex and rich datasets like Gutenberg. Models trained on the CHILDES and TinyStories datasets underperformed across all model sizes. These findings suggest that the optimal dataset for sample efficient training depends on the model size, and that neither child-directed speech nor simplified stories are optimal for language models of all sizes. We highlight the importance of considering both dataset composition and model capacity for effective sample efficient language model training.