Language Models Improve When Pretraining Data Matches Target Tasks

TL;DR

This paper introduces BETR, a data selection method that improves language model performance by explicitly matching pretraining data to target benchmarks, demonstrating significant gains across various scales and tasks.

Contribution

We propose BETR, a benchmark-targeted ranking method that aligns pretraining data with evaluation benchmarks, enhancing model performance and generalization.

Findings

BETR achieves 2.1x compute efficiency over baseline.

BETR improves performance on 9 out of 10 tasks.

Larger models require less aggressive data filtering.

Abstract

Every data selection method inherently has a target. In practice, these targets often emerge implicitly through benchmark-driven iteration: researchers develop selection strategies, train models, measure benchmark performance, then refine accordingly. This raises a natural question: what happens when we make this optimization explicit? To explore this, we propose benchmark-targeted ranking (BETR), a simple method that selects pretraining documents based on similarity to benchmark training examples. BETR embeds benchmark examples and a sample of pretraining documents in a shared space, scores this sample by similarity to benchmarks, then trains a lightweight classifier to predict these scores for the full corpus. We compare data selection methods by training over 500 models spanning $10^{19}$ to $10^{22}$ FLOPs and fitting scaling laws to them. From this, we find that simply aligning pretraining data to evaluation benchmarks using BETR achieves a 2.1x compute multiplier over DCLM-Baseline (4.7x over unfiltered data) and improves performance on 9 out of 10 tasks across all scales. BETR also generalizes well: when targeting a diverse set of benchmarks disjoint from our evaluation suite, it still matches or outperforms baselines. Our scaling analysis further reveals a clear trend: larger models require less aggressive filtering. Overall, our findings show that directly matching pretraining data to target tasks precisely shapes model capabilities and highlight that optimal selection strategies must adapt to model scale.