Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning

TL;DR

This paper presents a novel dataset distillation method that prunes non-beneficial samples using loss-based sampling before distillation, resulting in improved generalization and accuracy even with significant dataset reduction.

Contribution

Introduces a 'Prune First, Distill After' framework that enhances dataset distillation by systematically pruning datasets prior to distillation, improving performance and robustness.

Findings

Achieves up to 5.2% accuracy improvement with 80% dataset pruning.

Creates a representative core-set that generalizes across architectures.

Demonstrates robustness and effectiveness of the pruning-based distillation approach.

Abstract

Dataset distillation has gained significant interest in recent years, yet existing approaches typically distill from the entire dataset, potentially including non-beneficial samples. We introduce a novel "Prune First, Distill After" framework that systematically prunes datasets via loss-based sampling prior to distillation. By leveraging pruning before classical distillation techniques and generative priors, we create a representative core-set that leads to enhanced generalization for unseen architectures - a significant challenge of current distillation methods. More specifically, our proposed framework significantly boosts distilled quality, achieving up to a 5.2 percentage points accuracy increase even with substantial dataset pruning, i.e., removing 80% of the original dataset prior to distillation. Overall, our experimental results highlight the advantages of our easy-sample prioritization and cross-architecture robustness, paving the way for more effective and high-quality dataset distillation.