Generalizing Dataset Distillation via Deep Generative Prior

TL;DR

This paper introduces a novel dataset distillation method leveraging pre-trained deep generative models to synthesize data, significantly enhancing generalization across architectures and scalability to high-resolution datasets.

Contribution

The authors propose a new optimization algorithm that uses deep generative priors to improve dataset distillation, addressing generalization and scalability issues.

Findings

Improved cross-architecture generalization in dataset distillation.

Enhanced scalability to high-resolution datasets.

Augmentation of existing distillation techniques with generative priors.

Abstract

Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite recent progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.