Distilled Replay: Overcoming Forgetting through Synthetic Samples

TL;DR

This paper introduces Distilled Replay, a novel continual learning strategy that uses a small, distilled buffer of highly informative samples to effectively mitigate forgetting, reducing memory requirements.

Contribution

It proposes a distillation-based method to create a minimal yet highly informative replay buffer for continual learning.

Findings

Distilled Replay outperforms popular replay strategies on multiple benchmarks.

The method uses only 1 pattern per class in the buffer.

It effectively mitigates catastrophic forgetting with minimal memory.

Abstract

Replay strategies are Continual Learning techniques which mitigate catastrophic forgetting by keeping a buffer of patterns from previous experiences, which are interleaved with new data during training. The amount of patterns stored in the buffer is a critical parameter which largely influences the final performance and the memory footprint of the approach. This work introduces Distilled Replay, a novel replay strategy for Continual Learning which is able to mitigate forgetting by keeping a very small buffer (1 pattern per class) of highly informative samples. Distilled Replay builds the buffer through a distillation process which compresses a large dataset into a tiny set of informative examples. We show the effectiveness of our Distilled Replay against popular replay-based strategies on four Continual Learning benchmarks.