Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation

TL;DR

This paper introduces a two-stage method combining coreset selection and generative models to improve dataset distillation efficiency, quality, and diversity, achieving over 10% better results on large-scale benchmarks.

Contribution

It proposes a novel diffusion-augmented coreset expansion technique that enhances dataset distillation by dynamically expanding compressed coresets with generative models.

Findings

Over 10% improvement over state-of-the-art methods

Effective compression and expansion of datasets

Robust performance across multiple benchmarks

Abstract

With the rapid scaling of neural networks, data storage and communication demands have intensified. Dataset distillation has emerged as a promising solution, condensing information from extensive datasets into a compact set of synthetic samples by solving a bilevel optimization problem. However, current methods face challenges in computational efficiency, particularly with high-resolution data and complex architectures. Recently, knowledge-distillation-based dataset condensation approaches have made this process more computationally feasible. Yet, with the recent developments of generative foundation models, there is now an opportunity to achieve even greater compression, enhance the quality of distilled data, and introduce valuable diversity into the data representation. In this work, we propose a two-stage solution. First, we compress the dataset by selecting only the most informative patches to form a coreset. Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time, enhancing the resolution of these patches and introducing controlled variability to the coreset. Our extensive experiments demonstrate the robustness and efficiency of our approach across a range of dataset distillation benchmarks. We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks. The code will be released soon.