DsDm: Model-Aware Dataset Selection with Datamodels

TL;DR

This paper introduces DsDm, a model-aware dataset selection method that optimizes data subsets for training large models, leading to significant performance improvements over traditional quality-based filtering.

Contribution

It formulates dataset selection as an optimization problem considering the learning process, enabling more effective data subset choices for model training.

Findings

Double the performance with a 2x compute multiplier over baseline methods.

Model-aware selection outperforms traditional quality-based filtering.

Effective on both specific and unseen tasks.

Abstract

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in practice the opposite can often happen: we find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. To develop better methods for selecting data, we start by framing dataset selection as an optimization problem that we can directly solve for: given target tasks, a learning algorithm, and candidate data, select the subset that maximizes model performance. This framework thus avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks. Our resulting method greatly improves language model (LM) performance on both pre-specified tasks and previously unseen tasks. Specifically, choosing target tasks representative of standard LM problems and evaluating on diverse held-out benchmarks, our selected datasets provide a 2x compute multiplier over baseline methods.