Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

TL;DR

This paper investigates how neural network representations change during sequential task learning, revealing that deeper layers are key to catastrophic forgetting and that task similarity influences forgetting severity.

Contribution

It provides a detailed analysis of the role of hidden representations in catastrophic forgetting and proposes methods to mitigate it by stabilizing deeper layers.

Findings

Deeper layers are the primary source of forgetting.

Mitigation methods stabilize deeper layers.

Maximum forgetting occurs at intermediate task similarity.

Abstract

A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.