Effective Data Pruning through Score Extrapolation

TL;DR

This paper introduces a novel importance score extrapolation framework that predicts sample importance using minimal data, enabling efficient data pruning without full initial training, applicable across various datasets and training paradigms.

Contribution

The paper proposes a new importance score extrapolation method using k-nearest neighbors and graph neural networks to reduce training costs in data pruning.

Findings

Effective across multiple datasets and training paradigms

Reduces computational costs of data pruning techniques

Applicable to state-of-the-art pruning methods

Abstract

Training advanced machine learning models demands massive datasets, resulting in prohibitive computational costs. To address this challenge, data pruning techniques identify and remove redundant training samples while preserving model performance. Yet, existing pruning techniques predominantly require a full initial training pass to identify removable samples, negating any efficiency benefits for single training runs. To overcome this limitation, we introduce a novel importance score extrapolation framework that requires training on only a small subset of data. We present two initial approaches in this framework - k-nearest neighbors and graph neural networks - to accurately predict sample importance for the entire dataset using patterns learned from this minimal subset. We demonstrate the effectiveness of our approach for 2 state-of-the-art pruning methods (Dynamic Uncertainty and TDDS), 4 different datasets (CIFAR-10, CIFAR-100, Places-365, and ImageNet), and 3 training paradigms (supervised, unsupervised, and adversarial). Our results indicate that score extrapolation is a promising direction to scale expensive score calculation methods, such as pruning, data attribution, or other tasks.