Improved Distribution Matching for Dataset Condensation

TL;DR

This paper introduces a more efficient dataset condensation method based on distribution matching, addressing key shortcomings with novel techniques to outperform previous methods while reducing computational costs.

Contribution

The paper proposes a novel distribution matching approach with three techniques to improve efficiency and scalability in dataset condensation.

Findings

Outperforms previous optimization-based methods

Requires significantly fewer computational resources

Scales effectively to larger datasets and models

Abstract

Dataset Condensation aims to condense a large dataset into a smaller one while maintaining its ability to train a well-performing model, thus reducing the storage cost and training effort in deep learning applications. However, conventional dataset condensation methods are optimization-oriented and condense the dataset by performing gradient or parameter matching during model optimization, which is computationally intensive even on small datasets and models. In this paper, we propose a novel dataset condensation method based on distribution matching, which is more efficient and promising. Specifically, we identify two important shortcomings of naive distribution matching (i.e., imbalanced feature numbers and unvalidated embeddings for distance computation) and address them with three novel techniques (i.e., partitioning and expansion augmentation, efficient and enriched model sampling, and class-aware distribution regularization). Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources, thereby scaling data condensation to larger datasets and models. Extensive experiments demonstrate the effectiveness of our method. Codes are available at https://github.com/uitrbn/IDM