Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

TL;DR

This paper investigates the fundamental limitations of score-based data pruning algorithms, revealing their failure in high compression regimes and proposing calibration methods to improve their performance, highlighting the strength of random pruning.

Contribution

The paper provides theoretical and empirical analysis showing score-based pruning algorithms fail under high compression and introduces calibration protocols to enhance their effectiveness.

Findings

Score-based algorithms fail in high compression regimes.

Random pruning remains a strong baseline in high compression.

Calibration protocols improve pruning performance with randomization.

Abstract

Data pruning algorithms are commonly used to reduce the memory and computational cost of the optimization process. Recent empirical results reveal that random data pruning remains a strong baseline and outperforms most existing data pruning methods in the high compression regime, i.e., where a fraction of $30\%$ or less of the data is kept. This regime has recently attracted a lot of interest as a result of the role of data pruning in improving the so-called neural scaling laws; in [Sorscher et al.], the authors showed the need for high-quality data pruning algorithms in order to beat the sample power law. In this work, we focus on score-based data pruning algorithms and show theoretically and empirically why such algorithms fail in the high compression regime. We demonstrate ``No Free Lunch" theorems for data pruning and present calibration protocols that enhance the performance of existing pruning algorithms in this high compression regime using randomization.