LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning

TL;DR

LEAD is an efficient data selection framework for LLM instruction tuning that estimates sample utility within the training loop, greatly reducing computational costs while improving model performance.

Contribution

LEAD introduces a novel utility estimation method and a two-stage selection strategy that eliminate the need for costly model inference during data selection.

Findings

Outperforms state-of-the-art methods in diverse benchmarks.

Improves model performance by up to 10.8%.

Reduces training data usage by 97.5% and training time by 5-10x.

Abstract

Instruction tuning has emerged as a critical paradigm for improving the capabilities and alignment of large language models (LLMs). However, existing iterative model-aware data selection methods incur significant computational overhead, as they rely on repeatedly performing full-dataset model inference to estimate sample utility for subsequent training iterations, creating a fundamental efficiency bottleneck. In this paper, we propose LEAD, an efficient iterative data selection framework that accurately estimates sample utility entirely within the standard training loop, eliminating the need for costly additional model inference. At its core, LEAD introduces Instance-Level Dynamic Uncertainty (IDU), a theoretically grounded utility function combining instantaneous training loss, gradient-based approximation of loss changes, and exponential smoothing of historical loss signals. To further scale efficiently to large datasets, LEAD employs a two-stage, coarse-to-fine selection strategy, adaptively prioritizing informative clusters through a multi-armed bandit mechanism, followed by precise fine-grained selection of high-utility samples using IDU. Extensive experiments across four diverse benchmarks show that LEAD significantly outperforms state-of-the-art methods, improving average model performance by 6.1%-10.8% while using only 2.5% of the training data and reducing overall training time by 5-10x.