The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions

TL;DR

This paper reviews recent advances in dataset distillation, focusing on scalability, methodological innovations, and applications across diverse domains, aiming to improve efficiency and generalization in deep learning models.

Contribution

It provides a comprehensive survey of recent scalable dataset distillation methods, introduces new frameworks like SRe2L, and discusses challenges and emerging applications.

Findings

SRe2L framework enhances condensation efficiency and effectiveness.

Soft label strategies improve model accuracy significantly.

Lossless distillation maximizes compression while maintaining performance.

Abstract

Dataset distillation, which condenses large-scale datasets into compact synthetic representations, has emerged as a critical solution for training modern deep learning models efficiently. While prior surveys focus on developments before 2023, this work comprehensively reviews recent advances, emphasizing scalability to large-scale datasets such as ImageNet-1K and ImageNet-21K. We categorize progress into a few key methodologies: trajectory matching, gradient matching, distribution matching, scalable generative approaches, and decoupling optimization mechanisms. As a comprehensive examination of recent dataset distillation advances, this survey highlights breakthrough innovations: the SRe2L framework for efficient and effective condensation, soft label strategies that significantly enhance model accuracy, and lossless distillation techniques that maximize compression while maintaining performance. Beyond these methodological advancements, we address critical challenges, including robustness against adversarial and backdoor attacks, effective handling of non-IID data distributions. Additionally, we explore emerging applications in video and audio processing, multi-modal learning, medical imaging, and scientific computing, highlighting its domain versatility. By offering extensive performance comparisons and actionable research directions, this survey equips researchers and practitioners with practical insights to advance efficient and generalizable dataset distillation, paving the way for future innovations.