Dataset Distillation as Data Compression: A Rate-Utility Perspective

TL;DR

This paper introduces a joint rate-utility optimization approach for dataset distillation, effectively compressing datasets into fewer synthetic samples while maintaining utility, outperforming existing methods in compression and accuracy trade-offs.

Contribution

It proposes a novel joint optimization framework that balances data compression and utility, using a parameterized latent code approach and a new storage metric for fair comparison.

Findings

Achieves up to 170x greater compression on benchmark datasets.

Consistently outperforms existing distillation methods across various metrics.

Establishes better rate-utility trade-offs in dataset distillation.

Abstract

Driven by the ``scale-is-everything'' paradigm, modern machine learning increasingly demands ever-larger datasets and models, yielding prohibitive computational and storage requirements. Dataset distillation mitigates this by compressing an original dataset into a small set of synthetic samples, while preserving its full utility. Yet, existing methods either maximize performance under fixed storage budgets or pursue suitable synthetic data representations for redundancy removal, without jointly optimizing both objectives. In this work, we propose a joint rate-utility optimization method for dataset distillation. We parameterize synthetic samples as optimizable latent codes decoded by extremely lightweight networks. We estimate the Shannon entropy of quantized latents as the rate measure and plug any existing distillation loss as the utility measure, trading them off via a Lagrange multiplier. To enable fair, cross-method comparisons, we introduce bits per class (bpc), a precise storage metric that accounts for sample, label, and decoder parameter costs. On CIFAR-10, CIFAR-100, and ImageNet-128, our method achieves up to $170\times$ greater compression than standard distillation at comparable accuracy. Across diverse bpc budgets, distillation losses, and backbone architectures, our approach consistently establishes better rate-utility trade-offs.