Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning

TL;DR

This paper introduces Data-Distill-Net, a novel dataset distillation framework for continual learning that uses a learnable memory buffer and lightweight distillation to better retain knowledge and reduce forgetting.

Contribution

It proposes a new dataset distillation approach with a learnable memory buffer and lightweight module, improving continual learning performance over traditional replay methods.

Findings

Achieves competitive results on various datasets.

Effectively mitigates forgetting in continual learning.

Uses a lightweight distillation module to reduce computational overhead.

Abstract

Replay-based continual learning (CL) methods assume that models trained on a small subset can also effectively minimize the empirical risk of the complete dataset. These methods maintain a memory buffer that stores a sampled subset of data from previous tasks to consolidate past knowledge. However, this assumption is not guaranteed in practice due to the limited capacity of the memory buffer and the heuristic criteria used for buffer data selection. To address this issue, we propose a new dataset distillation framework tailored for CL, which maintains a learnable memory buffer to distill the global information from the current task data and accumulated knowledge preserved in the previous memory buffer. Moreover, to avoid the computational overhead and overfitting risks associated with parameterizing the entire buffer during distillation, we introduce a lightweight distillation module that can achieve global information distillation solely by generating learnable soft labels for the memory buffer data. Extensive experiments show that, our method can achieve competitive results and effectively mitigates forgetting across various datasets. The source code will be publicly available.