Eidetic Learning: an Efficient and Provable Solution to Catastrophic Forgetting

TL;DR

Eidetic Learning introduces a provably effective method for neural networks to learn multiple tasks sequentially without forgetting, using a capacity-partitioning approach that is efficient and easy to implement.

Contribution

The paper proposes Eidetic Learning, a novel approach that guarantees no catastrophic forgetting in neural networks without rehearsal, utilizing a task-routing mechanism similar to Mixture-of-Experts.

Findings

EideticNets are immune to catastrophic forgetting across various architectures and tasks.

The method is efficient with linear time and space complexity in the number of parameters.

Eidetic Learning is provably effective during both pre-training and fine-tuning phases.

Abstract

Catastrophic forgetting -- the phenomenon of a neural network learning a task t1 and losing the ability to perform it after being trained on some other task t2 -- is a long-standing problem for neural networks [McCloskey and Cohen, 1989]. We present a method, Eidetic Learning, that provably solves catastrophic forgetting. A network trained with Eidetic Learning -- here, an EideticNet -- requires no rehearsal or replay. We consider successive discrete tasks and show how at inference time an EideticNet automatically routes new instances without auxiliary task information. An EideticNet bears a family resemblance to the sparsely-gated Mixture-of-Experts layer Shazeer et al. [2016] in that network capacity is partitioned across tasks and the network itself performs data-conditional routing. An EideticNet is easy to implement and train, is efficient, and has time and space complexity linear in the number of parameters. The guarantee of our method holds for normalization layers of modern neural networks during both pre-training and fine-tuning. We show with a variety of network architectures and sets of tasks that EideticNets are immune to forgetting. While the practical benefits of EideticNets are substantial, we believe they can be benefit practitioners and theorists alike. The code for training EideticNets is available at https://github.com/amazon-science/eideticnet-training.