Dataset Distillation via Factorization

TL;DR

This paper introduces HaBa, a novel dataset distillation method that decomposes datasets into hallucination networks and bases, significantly improving data efficiency and classification accuracy while reducing parameters.

Contribution

The paper proposes a new dataset factorization approach, HaBa, which enhances dataset distillation by decomposing data into components, increasing informativeness and diversity, and improving downstream task performance.

Findings

Achieves up to 65% reduction in compressed parameters.

Improves classification accuracy by approximately 10% in cross-architecture tests.

Demonstrates significant performance gains over previous state-of-the-art methods.

Abstract

In this paper, we study \xw{dataset distillation (DD)}, from a novel perspective and introduce a \emph{dataset factorization} approach, termed \emph{HaBa}, which is a plug-and-play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, \emph{HaBa} explores decomposing a dataset into two components: data \emph{Ha}llucination networks and \emph{Ba}ses, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets. To furthermore increase the data efficiency of compression results, we further introduce a pair of adversarial contrastive constraints on the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization. Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65\%. Moreover, distilled datasets by our approach also achieve \textasciitilde10\% higher accuracy than baseline methods in cross-architecture generalization. Our code is available \href{https://github.com/Huage001/DatasetFactorization}{here}.