Efficient Dataset Distillation via Minimax Diffusion

TL;DR

This paper introduces a novel dataset distillation method using minimax diffusion techniques to generate representative and diverse surrogate datasets efficiently, significantly reducing computational costs while improving performance.

Contribution

The work integrates generative diffusion models with minimax criteria for dataset distillation, enabling faster and more effective surrogate data generation.

Findings

Achieves state-of-the-art validation performance.

Requires less than one-twentieth the time of previous methods.

Outperforms prior approaches in high IPC settings.

Abstract

Dataset distillation reduces the storage and computational consumption of training a network by generating a small surrogate dataset that encapsulates rich information of the original large-scale one. However, previous distillation methods heavily rely on the sample-wise iterative optimization scheme. As the images-per-class (IPC) setting or image resolution grows larger, the necessary computation will demand overwhelming time and resources. In this work, we intend to incorporate generative diffusion techniques for computing the surrogate dataset. Observing that key factors for constructing an effective surrogate dataset are representativeness and diversity, we design additional minimax criteria in the generative training to enhance these facets for the generated images of diffusion models. We present a theoretical model of the process as hierarchical diffusion control demonstrating the flexibility of the diffusion process to target these criteria without jeopardizing the faithfulness of the sample to the desired distribution. The proposed method achieves state-of-the-art validation performance while demanding much less computational resources. Under the 100-IPC setting on ImageWoof, our method requires less than one-twentieth the distillation time of previous methods, yet yields even better performance. Source code and generated data are available in https://github.com/vimar-gu/MinimaxDiffusion.