Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

TL;DR

This paper introduces an optimizer-aware online data selection and reweighting method for fine-tuning large language models, improving convergence and performance by considering optimizer states and data interactions.

Contribution

It proposes a novel Filter-then-Weight algorithm that dynamically filters and reweights data during online LLM fine-tuning, accounting for optimizer geometry and data redundancy.

Findings

Consistently improves convergence over existing methods.

Enhances downstream task performance under the same data budget.

Efficiently handles long-context data with optimized computations.

Abstract

Gradient-based data selection offers a principled framework for estimating sample utility in large language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the current optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.