EWGN: Elastic Weight Generation and Context Switching in Deep Learning

TL;DR

This paper introduces EWGN, a neural network architecture that dynamically generates task-specific weights for continual learning, aiming to reduce catastrophic forgetting through context switching and weight consolidation.

Contribution

The paper proposes Elastic Weight Generative Networks (EWGN), a novel architecture that enables dynamic, input-dependent weight generation for improved task retention in continual learning.

Findings

EWGN effectively retains previous task knowledge in vision datasets.

EWGN outperforms standard networks in continual learning scenarios.

Dynamic weight generation aids in mitigating catastrophic forgetting.

Abstract

The ability to learn and retain a wide variety of tasks is a hallmark of human intelligence that has inspired research in artificial general intelligence. Continual learning approaches provide a significant step towards achieving this goal. It has been known that task variability and context switching are challenging for learning in neural networks. Catastrophic forgetting refers to the poor performance on retention of a previously learned task when a new task is being learned. Switching between different task contexts can be a useful approach to mitigate the same by preventing the interference between the varying task weights of the network. This paper introduces Elastic Weight Generative Networks (EWGN) as an idea for context switching between two different tasks. The proposed EWGN architecture uses an additional network that generates the weights of the primary network dynamically while consolidating the weights learned. The weight generation is input-dependent and thus enables context switching. Using standard computer vision datasets, namely MNIST and fashion-MNIST, we analyse the retention of previously learned task representations in Fully Connected Networks, Convolutional Neural Networks, and EWGN architectures with Stochastic Gradient Descent and Elastic Weight Consolidation learning algorithms. Understanding dynamic weight generation and context-switching ability can be useful in enabling continual learning for improved performance.