Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks

TL;DR

This paper introduces a dataset compression method using shared memory bases and flexible addressing, enabling efficient re-training and continual learning with improved accuracy and reduced data size.

Contribution

It presents a novel dataset distillation approach that uses shared bases and learned addressing functions, achieving higher compression and better performance in re-training and continual learning.

Findings

State-of-the-art results on dataset distillation benchmarks.

Significant accuracy improvements on CIFAR10 and CIFAR100.

Enhanced continual learning performance across multiple benchmarks.

Abstract

We propose an algorithm that compresses the critical information of a large dataset into compact addressable memories. These memories can then be recalled to quickly re-train a neural network and recover the performance (instead of storing and re-training on the full original dataset). Building upon the dataset distillation framework, we make a key observation that a shared common representation allows for more efficient and effective distillation. Concretely, we learn a set of bases (aka ``memories'') which are shared between classes and combined through learned flexible addressing functions to generate a diverse set of training examples. This leads to several benefits: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with more effective distillation is achieved; and 3) more generalized queries are allowed beyond recalling the original classes. We demonstrate state-of-the-art results on the dataset distillation task across six benchmarks, including up to 16.5% and 9.7% in retained accuracy improvement when distilling CIFAR10 and CIFAR100 respectively. We then leverage our framework to perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY. The code is released on our project webpage https://github.com/princetonvisualai/RememberThePast-DatasetDistillation.