Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment

TL;DR

This paper investigates the causes of accuracy loss in continual learning, revealing that readout misalignment and representational shifts are major factors, with implications for improving neural network robustness and biological alignment.

Contribution

It identifies readout misalignment as a key factor in catastrophic forgetting and characterizes the representational changes involved.

Findings

Readout misalignment accounts for most accuracy loss.

Representational geometry is partially conserved despite shifts.

Representational changes scale with hidden layer dimensionality.

Abstract

Unlike primates, training artificial neural networks on changing data distributions leads to a rapid decrease in performance on old tasks. This phenomenon is commonly referred to as catastrophic forgetting. In this paper, we investigate the representational changes that underlie this performance decrease and identify three distinct processes that together account for the phenomenon. The largest component is a misalignment between hidden representations and readout layers. Misalignment occurs due to learning on additional tasks and causes internal representations to shift. Representational geometry is partially conserved under this misalignment and only a small part of the information is irrecoverably lost. All types of representational changes scale with the dimensionality of hidden representations. These insights have implications for deep learning applications that need to be continuously updated, but may also aid aligning ANN models to the rather robust biological vision.