Dataset Condensation with Latent Quantile Matching

TL;DR

This paper introduces Latent Quantile Matching (LQM), a novel dataset condensation method that improves distribution matching by aligning quantiles of latent embeddings, leading to better synthetic datasets for training machine learning models.

Contribution

The paper proposes LQM, addressing limitations of mean-based distribution matching by using quantile matching, and demonstrates its effectiveness on image, graph datasets, and continual graph learning.

Findings

LQM outperforms previous state-of-the-art distribution matching methods.

LQM improves continual graph learning performance.

LQM enhances memory efficiency and privacy in CGL.

Abstract

Dataset condensation (DC) methods aim to learn a smaller synthesized dataset with informative data records to accelerate the training of machine learning models. Current distribution matching (DM) based DC methods learn a synthesized dataset by matching the mean of the latent embeddings between the synthetic and the real dataset. However two distributions with the same mean can still be vastly different. In this work we demonstrate the shortcomings of using Maximum Mean Discrepancy to match latent distributions i.e. the weak matching power and lack of outlier regularization. To alleviate these shortcomings we propose our new method: Latent Quantile Matching (LQM) which matches the quantiles of the latent embeddings to minimize the goodness of fit test statistic between two distributions. Empirical experiments on both image and graph-structured datasets show that LQM matches or outperforms previous state of the art in distribution matching based DC. Moreover we show that LQM improves the performance in continual graph learning (CGL) setting where memory efficiency and privacy can be important. Our work sheds light on the application of DM based DC for CGL.