Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

TL;DR

This paper introduces a novel diversity-driven dataset distillation method that uses dynamic weight adjustment to produce more representative and diverse synthetic datasets, improving training efficiency and effectiveness.

Contribution

It proposes a new approach employing directed weight adjustment to enhance diversity in dataset synthesis, which is theoretically analyzed and empirically validated.

Findings

Improved diversity and representativeness of synthetic datasets

Enhanced training performance on multiple datasets

Reduced computational costs in dataset distillation

Abstract

The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.https://github.com/AngusDujw/Diversity-Driven-Synthesis.