Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models

TL;DR

This paper introduces a novel method for optimizing language model pre-training data mixtures using a Mixture of Data Experts, leading to improved performance on downstream tasks by better approximating model loss.

Contribution

It presents a new approach combining data expert models with regression to optimize data mixtures, enhancing language model training efficiency and effectiveness.

Findings

Significantly improved performance over baseline mixture approaches.

Effective optimization method improves few-shot downstream task results.

Theoretical insights support the aggregation of data expert predictions.

Abstract

We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE). We use this approximation as a source of additional features in a regression model, trained from observations of model loss for a small number of mixtures. Experiments with Transformer decoder-only language models in the range of 70M to 1B parameters on the SlimPajama dataset show that our method achieves significantly better performance than approaches that train regression models using only the mixture rates as input features. Combining this improved optimization method with an objective that takes into account cross-entropy on end task data leads to superior performance on few-shot downstream evaluations. We also provide theoretical insights on why aggregation of data expert predictions can provide good approximations to model losses for data mixtures.