How catastrophic can catastrophic forgetting be in linear regression?

TL;DR

This paper analyzes the extent of catastrophic forgetting in overparameterized linear models across sequential tasks, providing exact bounds and revealing connections to classical algorithms like the Kaczmarz method.

Contribution

It offers the first exact expressions and bounds for forgetting in linear models, linking continual learning to alternating projections and the Kaczmarz method, and explores effects of task ordering.

Findings

Upper bound of T^2 * min{1/√k, d/k} on forgetting in cyclic task presentation

Forgetting can be significantly reduced with random task ordering

Contrasts between forgetting and convergence to offline solutions in linear models

Abstract

To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the model forgets the true labels of earlier tasks after training on subsequent tasks, obtaining exact expressions and bounds. We establish connections between continual learning in the linear setting and two other research areas: alternating projections and the Kaczmarz method. In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas. In particular, when T tasks in d dimensions are presented cyclically for k iterations, we prove an upper bound of T^2 * min{1/sqrt(k), d/k} on the forgetting. This stands in contrast to the convergence to the offline solution, which can be arbitrarily slow according to existing alternating projection results. We further show that the T^2 factor can be lifted when tasks are presented in a random ordering.