The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

TL;DR

This paper presents an analytical model examining how task similarity and overparameterization jointly influence catastrophic forgetting in continual learning, revealing nuanced patterns and validating results with synthetic and neural network experiments.

Contribution

It introduces an exact analytical expression for forgetting in a two-task linear regression model considering both task similarity and overparameterization, highlighting their joint effects.

Findings

Intermediate task similarity causes most forgetting in highly overparameterized models.

Near the interpolation threshold, forgetting decreases monotonically with task similarity.

Validation with synthetic data and neural networks confirms the analytical predictions.

Abstract

In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contrast, our paper examines how task similarity and overparameterization jointly affect forgetting in an analyzable model. Specifically, we focus on two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task (an abstraction of random permutation tasks). We derive an exact analytical expression for the expected forgetting - and uncover a nuanced pattern. In highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases monotonically with the expected task similarity. We validate our findings with linear regression on synthetic data, and with neural networks on established permutation task benchmarks.