Diffusion Models as Dataset Distillation Priors

TL;DR

This paper introduces Diffusion As Priors (DAP), a training-free method that guides diffusion models to produce more representative and high-quality datasets for distillation, outperforming existing approaches.

Contribution

DAP formalizes the representativeness prior in diffusion models using feature space similarity and guides the reverse diffusion process without retraining.

Findings

DAP outperforms state-of-the-art methods on ImageNet-1K.

It achieves better cross-architecture generalization.

The method improves dataset quality without additional training.

Abstract

Dataset distillation aims to synthesize compact yet informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, generalization, and representativeness in a single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration of external constraints to enhance data quality. To address this, we propose Diffusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using a Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Extensive experiments on large-scale datasets, such as ImageNet-1K and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high-fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides a practical, training-free framework for improving the quality of the distilled dataset.