Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws

TL;DR

This paper introduces a unified theoretical framework for dataset distillation, establishing laws that describe how distilled dataset size relates to performance, configuration diversity, and method interchangeability, supported by empirical validation.

Contribution

It provides the first comprehensive theoretical analysis of dataset distillation, including scaling and coverage laws, and clarifies the interchangeability of different matching methods.

Findings

Error decreases with larger distilled datasets following a scaling law.

Required sample size scales linearly with configuration diversity.

Matching methods are interchangeable surrogates reducing generalization error.

Abstract

Dataset distillation (DD) aims to construct compact synthetic datasets that allow models to achieve comparable performance to full-data training while substantially reducing storage and computation. Despite rapid empirical progress, its theoretical foundations remain limited: existing methods (gradient, distribution, trajectory matching) are built on heterogeneous surrogate objectives and optimization assumptions, which makes it difficult to analyze their common principles or provide general guarantees. Moreover, it is still unclear under what conditions distilled data can retain the effectiveness of full datasets when the training configuration, such as optimizer, architecture, or augmentation, changes. To answer these questions, we propose a unified theoretical framework, termed configuration--dynamics--error analysis, which reformulates major DD approaches under a common generalization-error perspective and provides two main results: (i) a scaling law that provides a single-configuration upper bound, characterizing how the error decreases as the distilled sample size increases and explaining the commonly observed performance saturation effect; and (ii) a coverage law showing that the required distilled sample size scales linearly with configuration diversity, with provably matching upper and lower bounds. In addition, our unified analysis reveals that various matching methods are interchangeable surrogates, reducing the same generalization error, clarifying why they can all achieve dataset distillation and providing guidance on how surrogate choices affect sample efficiency and robustness. Experiments across diverse methods and configurations empirically confirm the derived laws, advancing a theoretical foundation for DD and enabling theory-driven design of compact, configuration-robust dataset distillation.