BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges

TL;DR

This paper introduces BWS, a universal data subset selection method that efficiently chooses the most informative samples across a wide range of selection ratios, improving training efficiency for large datasets.

Contribution

The paper proposes BWS, a novel method for selecting data subsets based on difficulty scores, effective across various selection ratios and datasets, with an evaluation mechanism using kernel ridge regression.

Findings

BWS outperforms baseline methods across multiple datasets.

Effective across broad selection ratios from easy to difficult samples.

Applicable to training from scratch and fine-tuning scenarios.

Abstract

Data subset selection aims to find a smaller yet informative subset of a large dataset that can approximate the full-dataset training, addressing challenges associated with training neural networks on large-scale datasets. However, existing methods tend to specialize in either high or low selection ratio regimes, lacking a universal approach that consistently achieves competitive performance across a broad range of selection ratios. We introduce a universal and efficient data subset selection method, Best Window Selection (BWS), by proposing a method to choose the best window subset from samples ordered based on their difficulty scores. This approach offers flexibility by allowing the choice of window intervals that span from easy to difficult samples. Furthermore, we provide an efficient mechanism for selecting the best window subset by evaluating its quality using kernel ridge regression. Our experimental results demonstrate the superior performance of BWS compared to other baselines across a broad range of selection ratios over datasets, including CIFAR-10/100 and ImageNet, and the scenarios involving training from random initialization or fine-tuning of pre-trained models.