From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning

TL;DR

This paper provides a spectral analysis of forgetting in continual learning by studying how the task distribution influences performance loss, offering a theoretical framework for understanding forgetting dynamics.

Contribution

It introduces a spectral characterization of forgetting based on task distribution, extending prior order-based analyses to a distributional perspective in linear models.

Findings

Derived an exact operator identity for forgetting.

Established an upper bound and asymptotic convergence rate.

Linked forgetting rate to geometric properties of the task distribution.

Abstract

A central challenge in continual learning is forgetting, the loss of performance on previously learned tasks induced by sequential adaptation to new ones. While forgetting has been extensively studied empirically, rigorous theoretical characterizations remain limited. A notable step in this direction is \citet{evron2022catastrophic}, which analyzes forgetting under random orderings of a fixed task collection in overparameterized linear regression. We shift the perspective from order to distribution. Rather than asking how a fixed task collection behaves under random orderings, we study an exact-fit linear regime in which tasks are sampled i.i.d.\ from a task distribution~$\Pi$, and ask how the generating distribution itself governs forgetting. In this setting, we derive an exact operator identity for the forgetting quantity, revealing a recursive spectral structure. Building on this identity, we establish an unconditional upper bound, identify the leading asymptotic term, and, in generic nondegenerate cases, characterize the convergence rate up to constants. We further relate this rate to geometric properties of the task distribution, clarifying what drives slow or fast forgetting in this model.