MIM4DD: Mutual Information Maximization for Dataset Distillation

TL;DR

MIM4DD introduces a mutual information maximization approach for dataset distillation, improving synthetic data quality by explicitly measuring shared information, and can enhance existing state-of-the-art methods.

Contribution

It proposes using mutual information as a metric for dataset distillation and develops a contrastive learning framework to maximize MI between real and synthetic datasets.

Findings

MIM4DD effectively enhances dataset distillation performance.

The method is compatible as an add-on to existing techniques.

Experimental results demonstrate improved test accuracy.

Abstract

Dataset distillation (DD) aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model. State-of-the-art (SoTA) methods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig.1, Left), such as gradients and training trajectories. DD is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets that share the same labels as positive pairs and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As a result, the targeted MI can be transformed into a lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that MIM4DD can be implemented as an add-on module to existing SoTA DD methods.