Optimal Splitting of Language Models from Mixtures to Specialized Domains

TL;DR

This paper introduces a method to optimally allocate compute resources for training multiple language models across various domains, improving performance on knowledge and reasoning tasks by predicting loss and scaling laws.

Contribution

It proposes a novel approach for independent pretraining and optimal compute allocation using scaling laws, enhancing multi-domain language model training efficiency.

Findings

Accurately predicts model loss with different pretraining and specialization tokens.

Improves performance on common sense knowledge and reasoning benchmarks.

Scales effectively to larger models and datasets.

Abstract

Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D' specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets.