MyGO: Memory Yielding Generative Offline-consolidation for Lifelong Learning Systems

TL;DR

MyGO is a lifelong learning framework that uses generative models to consolidate knowledge offline, reducing memory requirements and privacy concerns while effectively mitigating catastrophic forgetting across vision and language tasks.

Contribution

It introduces a biologically inspired wake-sleep cycle approach that consolidates knowledge with generative models, eliminating the need for raw data storage in lifelong learning.

Findings

Significantly reduces catastrophic forgetting in vision and language tasks.

Maintains high accuracy across multiple tasks without raw data storage.

Effective domain-general lifelong learning performance.

Abstract

Continual or Lifelong Learning aims to develop models capable of acquiring new knowledge from a sequence of tasks without catastrophically forgetting what has been learned before. Existing approaches often rely on storing samples from previous tasks (experience replay) or employing complex regularization terms to protect learned weights. However, these methods face challenges related to data privacy, storage limitations, and performance degradation when tasks are dissimilar. To address these challenges, we introduce MyGO (Memory Yielding Generative Offline-consolidation), a novel lifelong learning framework inspired by the biological wake-sleep cycle. During the "wake" phase, the system rapidly learns a new task and trains a compact generative model (Generative Memory, G-mem) to capture its data distribution. During the "sleep" phase, the system enters an offline state, using all learned G-mem models to generate pseudo-data ("dreams") and consolidate new and old knowledge into a core feature extractor via knowledge distillation. This approach obviates the need to store any raw data, retaining only compact generative models, which offers significant advantages in privacy and storage efficiency. We evaluate MyGO on computer vision (Split-MNIST) and natural language processing (Split-AG News) benchmarks, comparing it against a sequential fine-tuning baseline. The results demonstrate that MyGO significantly mitigates catastrophic forgetting and maintains high average accuracy across tasks, proving the framework's effectiveness and domain-generality.