Scaling Parameter-Constrained Language Models with Quality Data

TL;DR

This paper extends language model scaling laws by incorporating data quality metrics, specifically effective training tokens based on diversity and syntheticity, to better predict model performance.

Contribution

It introduces a new formulation of effective training tokens that accounts for data quality, enhancing the understanding of scaling laws in parameter-constrained language models.

Findings

Effective training tokens correlate with model accuracy (Pearson +0.83).

Data quality metrics improve predictions of model performance.

Analysis of data sampling and synthesis techniques on data quality impact.

Abstract

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation -- effective training tokens -- which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over $200$ models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.