Delving into Effective Gradient Matching for Dataset Condensation

TL;DR

This paper investigates the principles of gradient matching in dataset condensation, proposing multi-level gradient matching and adaptive strategies to improve synthetic dataset quality and training efficiency.

Contribution

It introduces a comprehensive analysis of gradient matching, proposing multi-level gradient matching and an adaptive learning step strategy for better dataset condensation.

Findings

Enhanced accuracy over prior methods

Improved efficiency and generalization

Effective delay of overfitting through angle-focused distance

Abstract

As deep learning models and datasets rapidly scale up, network training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.