Overcoming Catastrophic Forgetting with Gaussian Mixture Replay

TL;DR

Gaussian Mixture Replay (GMR) is a novel continual learning method that uses Gaussian Mixture Models to generate samples, estimate density, and classify, effectively reducing catastrophic forgetting with low memory and constant training complexity.

Contribution

GMR introduces a unified GMM-based approach for rehearsal, enabling sample generation, density estimation, and classification in a single, memory-efficient network suitable for lifelong learning.

Findings

GMR effectively mitigates catastrophic forgetting in image classification tasks.

It operates with constant time complexity regardless of the number of sub-tasks.

GMR can detect task boundaries using GMM density estimation.

Abstract

We present Gaussian Mixture Replay (GMR), a rehearsal-based approach for continual learning (CL) based on Gaussian Mixture Models (GMM). CL approaches are intended to tackle the problem of catastrophic forgetting (CF), which occurs for Deep Neural Networks (DNNs) when sequentially training them on successive sub-tasks. GMR mitigates CF by generating samples from previous tasks and merging them with current training data. GMMs serve several purposes here: sample generation, density estimation (e.g., for detecting outliers or recognizing task boundaries) and providing a high-level feature representation for classification. GMR has several conceptual advantages over existing replay-based CL approaches. First of all, GMR achieves sample generation, classification and density estimation in a single network structure with strongly reduced memory requirements. Secondly, it can be trained at constant time complexity w.r.t. the number of sub-tasks, making it particularly suitable for life-long learning. Furthermore, GMR minimizes a differentiable loss function and seems to avoid mode collapse. In addition, task boundaries can be detected by applying GMM density estimation. Lastly, GMR does not require access to sub-tasks lying in the future for hyper-parameter tuning, allowing CL under real-world constraints. We evaluate GMR on multiple image datasets, which are divided into class-disjoint sub-tasks.