DataRater: Meta-Learned Dataset Curation

TL;DR

DataRater introduces a meta-learning approach to automatically evaluate and select valuable training data, significantly enhancing training efficiency for foundation models across various datasets and scales.

Contribution

It presents a novel meta-learning method using meta-gradients to estimate data value, enabling scalable and fine-grained dataset curation.

Findings

Improves training compute efficiency substantially.

Effective across different model sizes and datasets.

Outperforms manual data filtering methods.

Abstract

The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.