Rethinking Large-scale Dataset Compression: Shifting Focus From Labels to Images

TL;DR

This paper introduces a benchmark and a new dataset compression framework called PCA that focus on image data and hard labels, revealing that random subsets can perform competitively and challenging the reliance on soft labels in large-scale dataset distillation.

Contribution

The paper presents a benchmark for fair comparison of dataset compression methods and proposes PCA, a framework emphasizing image data and hard labels, achieving state-of-the-art results.

Findings

Random subsets perform competitively in dataset distillation.

Overreliance on soft labels may overlook the value of image data.

PCA achieves state-of-the-art performance focusing on images and hard labels.

Abstract

Dataset distillation and dataset pruning are two prominent techniques for compressing datasets to improve computational and storage efficiency. Despite their overlapping objectives, these approaches are rarely compared directly. Even within each field, the evaluation protocols are inconsistent across various methods, which complicates fair comparisons and hinders reproducibility. Considering these limitations, we introduce in this paper a benchmark that equitably evaluates methodologies across both distillation and pruning literatures. Notably, our benchmark reveals that in the mainstream dataset distillation setting for large-scale datasets, which heavily rely on soft labels from pre-trained models, even randomly selected subsets can achieve surprisingly competitive performance. This finding suggests that an overemphasis on soft labels may be diverting attention from the intrinsic value of the image data, while also imposing additional burdens in terms of generation, storage, and application. To address these issues, we propose a new framework for dataset compression, termed Prune, Combine, and Augment (PCA), which focuses on leveraging image data exclusively, relies solely on hard labels for evaluation, and achieves state-of-the-art performance in this setup. By shifting the emphasis back to the images, our benchmark and PCA framework pave the way for more balanced and accessible techniques in dataset compression research. Our code is available at: https://github.com/ArmandXiao/Rethinking-Dataset-Compression