Condensed Composite Memory Continual Learning

TL;DR

This paper introduces a novel continual learning method that synthesizes representative examples from datasets using shared components, significantly reducing memory requirements and mitigating catastrophic forgetting in deep neural networks.

Contribution

It proposes a new approach to rehearsal-based continual learning by learning weighted combinations of shared components for synthetic examples, improving memory efficiency and performance.

Findings

Outperforms existing methods on standard datasets

Requires fewer stored examples for effective learning

Reduces catastrophic forgetting significantly

Abstract

Deep Neural Networks (DNNs) suffer from a rapid decrease in performance when trained on a sequence of tasks where only data of the most recent task is available. This phenomenon, known as catastrophic forgetting, prevents DNNs from accumulating knowledge over time. Overcoming catastrophic forgetting and enabling continual learning is of great interest since it would enable the application of DNNs in settings where unrestricted access to all the training data at any time is not always possible, e.g. due to storage limitations or legal issues. While many recently proposed methods for continual learning use some training examples for rehearsal, their performance strongly depends on the number of stored examples. In order to improve performance of rehearsal for continual learning, especially for a small number of stored examples, we propose a novel way of learning a small set of synthetic examples which capture the essence of a complete dataset. Instead of directly learning these synthetic examples, we learn a weighted combination of shared components for each example that enables a significant increase in memory efficiency. We demonstrate the performance of our method on commonly used datasets and compare it to recently proposed related methods and baselines.