Overcoming catastrophic forgetting in neural networks

TL;DR

This paper evaluates Elastic Weight Consolidation (EWC) for continual learning, demonstrating its effectiveness in reducing forgetting across multiple tasks while analyzing its trade-offs and generalization capabilities.

Contribution

The study replicates and extends previous work on EWC, providing systematic comparisons and insights into its performance and hyperparameter effects in supervised learning.

Findings

EWC significantly reduces catastrophic forgetting compared to naive training.

EWC slightly compromises learning efficiency on new tasks.

Dropout and hyperparameters influence EWC's generalization.

Abstract

Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weight Consolidation, a regularization-based approach inspired by synaptic consolidation in biological neural systems, has been used to overcome this problem. In this study prior research is replicated and extended by evaluating EWC in supervised learning settings using the PermutedMNIST and RotatedMNIST benchmarks. Through systematic comparisons with L2 regularization and stochastic gradient descent (SGD) without regularization, we analyze how different approaches balance knowledge retention and adaptability. Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training while slightly compromising learning efficiency on new tasks. Moreover, we investigate the impact of dropout regularization and varying hyperparameters, offering insights into the generalization of EWC across diverse learning scenarios. These results underscore EWC's potential as a viable solution for lifelong learning in neural networks.