Dataset Distillation using Neural Feature Regression

TL;DR

This paper introduces FRePo, a neural feature regression method for dataset distillation that reduces memory and computation costs while achieving state-of-the-art performance on multiple image datasets.

Contribution

The paper proposes a novel neural feature regression approach with pooling for efficient dataset distillation, significantly improving speed and memory usage over prior methods.

Findings

FRePo achieves state-of-the-art results on CIFAR100, Tiny ImageNet, and ImageNet-1K.

The method reduces memory requirements by an order of magnitude.

Distilled data enhances downstream tasks like continual learning and membership inference defense.

Abstract

Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achieving the state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. The proposed algorithm is analogous to truncated backpropagation through time with a pool of models to alleviate various types of overfitting in dataset distillation. FRePo significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and ImageNet-1K. Furthermore, we show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense. Please check out our webpage at https://sites.google.com/view/frepo.