Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

TL;DR

This paper introduces ADAPT, an online reweighting framework for LLM training that dynamically adjusts sample importance during training, leading to better generalization than traditional offline data curation methods.

Contribution

The paper proposes ADAPT, a novel online reweighting approach that improves LLM training by dynamically adjusting sample importance without altering data size.

Findings

ADAPT outperforms offline data curation methods in experiments.

ADAPT achieves stronger cross-benchmark generalization.

Online reweighting reduces engineering overhead and brittleness.

Abstract

Data curation is a critical yet under-explored area in large language model (LLM) training. Existing methods, such as data selection and mixing, operate in an offline paradigm, detaching themselves from training. This separation introduces engineering overhead and makes the curation brittle: the entire pipeline must be re-run under model/task shifts. Moreover, offline methods alter data size through hard filtering or resampling, often sacrificing data diversity and harming generalization. We propose to rethink data curation as an online reweighting problem, where sample importance is dynamically adjusted during training via loss weighting rather than static pre-processing. Specifically, we introduce ADAPT (Adaptive Data reweighting for Pretraining and FineTuning), a dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity-based quality signals, without changing the number of training samples. Unlike offline methods that enforce a static data distribution, ADAPT acts as an implicit curriculum learner, progressively shifting focus from coarse-grained patterns to fine-grained semantic distinctions as the model evolves. Experiments on both instruction tuning and large-scale pretraining show that ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross-benchmark generalization under equal FLOPs.