M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy

TL;DR

M3D introduces a novel dataset condensation method that minimizes the maximum mean discrepancy in a reproducing kernel Hilbert space, effectively aligning higher-order distribution moments and outperforming state-of-the-art optimization-based methods on ImageNet.

Contribution

The paper proposes M3D, a distribution-matching approach that aligns all distribution moments for dataset condensation, surpassing existing methods in efficiency and performance.

Findings

M3D outperforms SOTA optimization-based methods on ImageNet.

Embedding distributions in RKHS enables higher-order moment alignment.

Extensive experiments verify the effectiveness of M3D.

Abstract

Training state-of-the-art (SOTA) deep models often requires extensive data, resulting in substantial training and storage costs. To address these challenges, dataset condensation has been developed to learn a small synthetic set that preserves essential information from the original large-scale dataset. Nowadays, optimization-oriented methods have been the primary method in the field of dataset condensation for achieving SOTA results. However, the bi-level optimization process hinders the practical application of such methods to realistic and larger datasets. To enhance condensation efficiency, previous works proposed Distribution-Matching (DM) as an alternative, which significantly reduces the condensation cost. Nonetheless, current DM-based methods still yield less comparable results to SOTA optimization-oriented methods. In this paper, we argue that existing DM-based methods overlook the higher-order alignment of the distributions, which may lead to sub-optimal matching results. Inspired by this, we present a novel DM-based method named M3D for dataset condensation by Minimizing the Maximum Mean Discrepancy between feature representations of the synthetic and real images. By embedding their distributions in a reproducing kernel Hilbert space, we align all orders of moments of the distributions of real and synthetic images, resulting in a more generalized condensed set. Notably, our method even surpasses the SOTA optimization-oriented method IDC on the high-resolution ImageNet dataset. Extensive analysis is conducted to verify the effectiveness of the proposed method. Source codes are available at https://github.com/Hansong-Zhang/M3D.