Dataset Distillation in Latent Space

TL;DR

This paper introduces a novel dataset distillation approach in latent space using autoencoders, reducing computational costs and increasing info-compactness while maintaining performance, enabling efficient distillation of high-resolution datasets.

Contribution

The work pioneers moving dataset distillation from pixel space to latent space, significantly improving efficiency and scalability of existing DD algorithms.

Findings

Reduced time and space complexity in DD methods

Achieved comparable performance to pixel-based methods

Enabled distillation of high-resolution datasets

Abstract

Dataset distillation (DD) is a newly emerging research area aiming at alleviating the heavy computational load in training models on large datasets. It tries to distill a large dataset into a small and condensed one so that models trained on the distilled dataset can perform comparably with those trained on the full dataset when performing downstream tasks. Among the previous works in this area, there are three key problems that hinder the performance and availability of the existing DD methods: high time complexity, high space complexity, and low info-compactness. In this work, we simultaneously attempt to settle these three problems by moving the DD processes from conventionally used pixel space to latent space. Encoded by a pretrained generic autoencoder, latent codes in the latent space are naturally info-compact representations of the original images in much smaller sizes. After transferring three mainstream DD algorithms to latent space, we significantly reduce time and space consumption while achieving similar performance, allowing us to distill high-resolution datasets or target at greater data ratio that previous methods have failed. Besides, within the same storage budget, we can also quantitatively deliver more latent codes than pixel-level images, which further boosts the performance of our methods.