Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks

TL;DR

This paper introduces Pseudo-Recursal, a method using GANs to generate representative items for pseudo-rehearsal, effectively mitigating catastrophic forgetting in deep neural networks across complex tasks like CIFAR-10, SVHN, and MNIST.

Contribution

It presents a novel pseudo-rehearsal approach employing GANs to generate training data, enabling neural networks to learn sequentially without significant forgetting on complex datasets.

Findings

Only 1.67% accuracy loss on CIFAR-10 after sequential learning.

Gains 0.24% accuracy on SVHN after training.

Outperforms previous state-of-the-art methods in continual learning.

Abstract

In general, neural networks are not currently capable of learning tasks in a sequential fashion. When a novel, unrelated task is learnt by a neural network, it substantially forgets how to solve previously learnt tasks. One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task/s. This is very effective for simple tasks. However, pseudo-rehearsal has not yet been successfully applied to very complex tasks because in these tasks it is difficult to generate representative items. We accomplish pseudo-rehearsal by using a Generative Adversarial Network to generate items so that our deep network can learn to sequentially classify the CIFAR-10, SVHN and MNIST datasets. After training on all tasks, our network loses only 1.67% absolute accuracy on CIFAR-10 and gains 0.24% absolute accuracy on SVHN. Our model's performance is a substantial improvement compared to the current state of the art solution.