Accelerating Dataset Distillation via Model Augmentation

TL;DR

This paper introduces model augmentation techniques to accelerate dataset distillation, significantly reducing training time while maintaining high performance.

Contribution

It proposes using early-stage models and parameter perturbation to improve dataset distillation efficiency and effectiveness.

Findings

Achieves up to 20x speedup in dataset distillation

Maintains comparable performance to state-of-the-art methods

Reduces computational cost significantly

Abstract

Dataset Distillation (DD), a newly emerging field, aims at generating much smaller but efficient synthetic training datasets from large ones. Existing DD methods based on gradient matching achieve leading performance; however, they are extremely computationally intensive as they require continuously optimizing a dataset among thousands of randomly initialized models. In this paper, we assume that training the synthetic data with diverse models leads to better generalization performance. Thus we propose two model augmentation techniques, i.e. using early-stage models and parameter perturbation to learn an informative synthetic set with significantly reduced training cost. Extensive experiments demonstrate that our method achieves up to 20x speedup and comparable performance on par with state-of-the-art methods.