Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

TL;DR

This paper introduces Mimic Score, a geometry-based data utility metric that improves data selection efficiency by leveraging model weights, leading to faster training and better performance in image and CLIP models.

Contribution

The paper proposes Mimic Score and Grad-Mimic, novel methods that use readily available model weights for efficient data selection without needing validation datasets or influence computations.

Findings

Improves data efficiency and accelerates convergence across six image datasets.

Enables CLIP models to reduce training steps by 20.7%.

Enhances existing data filtering techniques with better performance.

Abstract

Large-scale web-crawled datasets contain noise, bias, and irrelevant information, necessitating data selection techniques. Existing methods depend on hand-crafted heuristics, downstream datasets, or require expensive influence-based computations -- all of which limit scalability and introduce unwanted data dependencies. To address this, we introduce the Mimic Score, a simple and geometry-based data-quality metric that evaluates utility by measuring alignment between a sample's gradients and a target direction induced by a pre-trained reference model. This leverages readily available model weights, avoids needing validation datasets, and incurs minimal computational overheads. Building on this metric, we propose Grad-Mimic, a two-stage framework that re-weights samples online to accelerate training and aggregates sample utilities offline to construct effective data filters. Empirically, we show that using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7% fewer training steps. Additionally, mimic score-based filters augment existing filtering techniques, enabling improved CLIP models trained with 4.7 million fewer samples.