Dataset Distillation

TL;DR

This paper introduces dataset distillation, a method to synthesize a small, synthetic dataset that enables training models to achieve performance comparable to using large datasets, significantly reducing data size.

Contribution

It proposes a novel dataset distillation technique that compresses large datasets into small synthetic sets, maintaining model performance with minimal data.

Findings

Successfully compresses 60,000 MNIST images into 10 synthetic images.

Achieves near-original performance with few gradient steps.

Outperforms alternative data reduction methods.

Abstract

Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.