What is Dataset Distillation Learning?

TL;DR

This paper investigates the nature of dataset distillation, revealing its limitations, how it retains information, and providing a framework for interpreting the semantic content of synthetic data.

Contribution

It offers new insights into the behavior, representativeness, and information content of distilled data, and introduces a framework for interpretation.

Findings

Distilled data cannot replace real data outside standard training.

Distillation retains information related to early training dynamics.

Individual distilled points contain meaningful semantic information.

Abstract

Dataset distillation has emerged as a strategy to overcome the hurdles associated with large datasets by learning a compact set of synthetic data that retains essential information from the original dataset. While distilled data can be used to train high performing models, little is understood about how the information is stored. In this study, we posit and answer three questions about the behavior, representativeness, and point-wise information content of distilled data. We reveal distilled data cannot serve as a substitute for real data during training outside the standard evaluation setting for dataset distillation. Additionally, the distillation process retains high task performance by compressing information related to the early training dynamics of real models. Finally, we provide an framework for interpreting distilled data and reveal that individual distilled data points contain meaningful semantic information. This investigation sheds light on the intricate nature of distilled data, providing a better understanding on how they can be effectively utilized.