Replay Can Provably Increase Forgetting

TL;DR

This paper provides a theoretical and empirical analysis of sample replay in continual learning, revealing that replay can sometimes increase forgetting and that its effectiveness depends on task relationships and sample selection.

Contribution

The work offers the first theoretical analysis showing that replay can be harmful and non-monotonic, and demonstrates this phenomenon in both linear models and neural networks.

Findings

Replay can increase forgetting even with more samples.

The effectiveness of replay depends on task relationships and sample selection.

Harmful replay behavior is observed in neural networks, not just linear models.

Abstract

Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced. One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting by retaining some examples from old tasks and including them in new training episodes. In this work, we provide a theoretical analysis of sample replay in an over-parameterized continual linear regression setting, where each task is given by a linear subspace and with enough replay samples, one would be able to eliminate forgetting. Our analysis focuses on sample replay and highlights the role of the replayed samples and the relationship between task subspaces. Surprisingly, we find that, even in a noiseless setting, forgetting can be non-monotonic with respect to the number of replay samples. We present tasks where replay can be harmful with respect to worst-case settings, and also in distributional settings where replay of randomly selected samples increases forgetting in expectation. We also give empirical evidence that harmful replay is not limited to training with linear models by showing similar behavior for a neural networks equipped with SGD. Through experiments on a commonly used benchmark, we provide additional evidence that, even in seemingly benign scenarios, performance of the replay heavily depends on the choice of replay samples and the relationship between tasks.