UDD: Dataset Distillation via Mining Underutilized Regions

TL;DR

This paper introduces UDD, a novel dataset distillation method that identifies and exploits underutilized regions in synthetic images to improve dataset efficiency and model performance across multiple datasets.

Contribution

UDD proposes dynamic policies to search and utilize underutilized regions in synthetic data, enhancing dataset utilization and model accuracy in dataset distillation.

Findings

Outperforms state-of-the-art on MNIST, FashionMNIST, SVHN, CIFAR-10, CIFAR-100

Improves CIFAR-10 and CIFAR-100 accuracy by 4.0% and 3.7% respectively

Uses category-wise feature contrastive loss to enhance class distinguishability

Abstract

Dataset distillation synthesizes a small dataset such that a model trained on this set approximates the performance of the original dataset. Recent studies on dataset distillation focused primarily on the design of the optimization process, with methods such as gradient matching, feature alignment, and training trajectory matching. However, little attention has been given to the issue of underutilized regions in synthetic images. In this paper, we propose UDD, a novel approach to identify and exploit the underutilized regions to make them informative and discriminate, and thus improve the utilization of the synthetic dataset. Technically, UDD involves two underutilized regions searching policies for different conditions, i.e., response-based policy and data jittering-based policy. Compared with previous works, such two policies are utilization-sensitive, equipping with the ability to dynamically adjust the underutilized regions during the training process. Additionally, we analyze the current model optimization problem and design a category-wise feature contrastive loss, which can enhance the distinguishability of different categories and alleviate the shortcomings of the existing multi-formation methods. Experimentally, our method improves the utilization of the synthetic dataset and outperforms the state-of-the-art methods on various datasets, such as MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100. For example, the improvements on CIFAR-10 and CIFAR-100 are 4.0\% and 3.7\% over the next best method with IPC=1, by mining the underutilized regions.