Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities

TL;DR

This paper investigates how the scale of rehearsal impacts continual learning, revealing that larger rehearsal does not always enhance memory and can even hinder adaptability, with insights validated through neural network experiments.

Contribution

It provides a theoretical framework analyzing rehearsal scale effects in continual learning, uncovering counterintuitive impacts on adaptability and memory retention.

Findings

Rehearsal can impair model adaptability.

Increasing rehearsal scale does not always improve memory.

Memory error exhibits a diminishing lower bound under certain conditions.

Abstract

Rehearsal is one of the key techniques for mitigating catastrophic forgetting and has been widely adopted in continual learning algorithms due to its simplicity and practicality. However, the theoretical understanding of how rehearsal scale influences learning dynamics remains limited. To address this gap, we formulate rehearsal-based continual learning as a multidimensional effectiveness-driven iterative optimization problem, providing a unified characterization across diverse performance metrics. Within this framework, we derive a closed-form analysis of adaptability, memorability, and generalization from the perspective of rehearsal scale. Our results uncover several intriguing and counterintuitive findings. First, rehearsal can impair model's adaptability, in sharp contrast to its traditionally recognized benefits. Second, increasing the rehearsal scale does not necessarily improve memory retention. When tasks are similar and noise levels are low, the memory error exhibits a diminishing lower bound. Finally, we validate these insights through numerical simulations and extended analyses on deep neural networks across multiple real-world datasets, revealing statistical patterns of rehearsal mechanisms in continual learning.