Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

TL;DR

This paper introduces a novel dataset pruning method that identifies important samples early in training using example difficulty and prediction uncertainty, significantly reducing pruning time while maintaining state-of-the-art accuracy.

Contribution

The paper proposes the DUAL score and ratio-adaptive sampling to enable efficient dataset pruning without full training, outperforming existing methods in speed and accuracy.

Findings

Reduces pruning time to 66% on ImageNet-1k with 60% accuracy at 90% pruning

Achieves 15% pruning time on CIFAR datasets with state-of-the-art performance

Maintains high accuracy despite aggressive dataset reduction

Abstract

Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.