OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

TL;DR

OPUS introduces a dynamic data selection method for large language model pre-training that leverages optimizer-induced updates, improving efficiency and performance across various settings with minimal additional computational cost.

Contribution

OPUS is a novel, optimizer-aware data selection framework that enhances pre-training efficiency by projecting candidate updates onto a stable target direction, outperforming static filters and full data training.

Findings

Outperforms baselines in GPT-2 pre-training on diverse corpora.

Achieves comparable or better results than full 200B-token training with fewer tokens.

Improves data efficiency in domain-specific continued pre-training.

Abstract

As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7\% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. In pre-training of GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B tokens, OPUS outperforms industrial-level baselines and even full 200B-token training. Moreover, when combined with industrial-level static filters, OPUS further improves pre-training efficiency, even with lower-quality data. Furthermore, in continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains in specialized domains.