Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning

TL;DR

This paper introduces a lightweight, class-agnostic regularizer called Information Maximization (IM) that enhances rehearsal-based continual learning by reducing forgetting and improving convergence across diverse datasets, including video data.

Contribution

The paper proposes a novel, minimal-overhead regularizer for rehearsal-based continual learning that is class-agnostic and effective across multiple domains and data types.

Findings

Consistently improves baseline performance across datasets and tasks.

Reduces forgetting with minimal computational overhead.

Enhances video continual learning performance despite domain differences.

Abstract

Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task. This issue arises due to the model's tendency to overwrite previously acquired knowledge with new information. We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches. We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods, which is based exclusively on the expected label distribution, thus making it class-agnostic. As a consequence, IM regularizer can be directly integrated into various rehearsal-based continual learning methods, reducing forgetting and favoring faster convergence. Our empirical validation shows that, across datasets and regardless of the number of tasks, our proposed regularization strategy consistently improves baseline performance at the expense of a minimal computational overhead. The lightweight nature of IM ensures that it remains a practical and scalable solution, making it applicable to real-world continual learning scenarios where efficiency is paramount. Finally, we demonstrate the data-agnostic nature of our regularizer by applying it to video data, which presents additional challenges due to its temporal structure and higher memory requirements. Despite the significant domain gap, our experiments show that IM regularizer also improves the performance of video continual learning methods.