Data-dependent and Oracle Bounds on Forgetting in Continual Learning

TL;DR

This paper develops theoretical bounds on forgetting in exemplar-free continual learning, providing practical tools to quantify and minimize knowledge loss across tasks.

Contribution

It introduces data-dependent and oracle bounds applicable to various models and algorithms, along with an algorithm based on these bounds.

Findings

Bounds are tight and practical for multiple continual learning scenarios.

The approach applies regardless of specific models or algorithms used.

Empirical results validate the effectiveness of the bounds and the derived algorithm.

Abstract

In continual learning, knowledge must be preserved and re-used between tasks, maintaining good transfer to future tasks and minimizing forgetting of previously learned ones. While several practical algorithms have been devised for this setting, there have been few theoretical works aiming to quantify and bound the degree of Forgetting in general settings. For \emph{exemplar-free} methods, we provide both data-dependent upper bounds that apply \emph{regardless of model and algorithm choice}, and oracle bounds for Gibbs posteriors. We derive an algorithm based on our bounds and demonstrate empirically that our approach yields tight and practical bounds on forgetting for several continual learning problems and algorithms.