On robustness of generative representations against catastrophic forgetting

TL;DR

This paper investigates why generative models are more robust against catastrophic forgetting than discriminative models, highlighting their potential advantages for continual learning.

Contribution

It provides empirical evidence comparing the robustness of generative and discriminative models, revealing the superior stability of generative representations.

Findings

Generative models are less prone to catastrophic forgetting than discriminative models.

Representations learned by discriminative models are more susceptible to forgetting.

The study suggests potential for using generative models beyond replay in continual learning.

Abstract

Catastrophic forgetting of previously learned knowledge while learning new tasks is a widely observed limitation of contemporary neural networks. Although many continual learning methods are proposed to mitigate this drawback, the main question remains unanswered: what is the root cause of catastrophic forgetting? In this work, we aim at answering this question by posing and validating a set of research hypotheses related to the specificity of representations built internally by neural models. More specifically, we design a set of empirical evaluations that compare the robustness of representations in discriminative and generative models against catastrophic forgetting. We observe that representations learned by discriminative models are more prone to catastrophic forgetting than their generative counterparts, which sheds new light on the advantages of developing generative models for continual learning. Finally, our work opens new research pathways and possibilities to adopt generative models in continual learning beyond mere replay mechanisms.