Leveraging Lightweight Generators for Memory Efficient Continual Learning

TL;DR

This paper introduces lightweight generators based on SVD to reduce memory usage in continual learning, significantly improving accuracy while maintaining minimal memory footprint and computational cost.

Contribution

It proposes a novel SVD-based generator approach that enhances existing continual learning methods with minimal memory and no additional training time.

Findings

Significant accuracy improvements over baseline methods.

Minimal memory footprint with effective distribution capture.

Single linear-time fitting step suffices for generator setup.

Abstract

Catastrophic forgetting can be trivially alleviated by keeping all data from previous tasks in memory. Therefore, minimizing the memory footprint while maximizing the amount of relevant information is crucial to the challenge of continual learning. This paper aims to decrease required memory for memory-based continuous learning algorithms. We explore the options of extracting a minimal amount of information, while maximally alleviating forgetting. We propose the usage of lightweight generators based on Singular Value Decomposition to enhance existing continual learning methods, such as A-GEM and Experience Replay. These generators need a minimal amount of memory while being maximally effective. They require no training time, just a single linear-time fitting step, and can capture a distribution effectively from a small number of data samples. Depending on the dataset and network architecture, our results show a significant increase in average accuracy compared to the original methods. Our method shows great potential in minimizing the memory footprint of memory-based continual learning algorithms.