Continual Learning Through Synaptic Intelligence

TL;DR

This paper introduces synaptic intelligence, a method inspired by biological neural networks, enabling artificial networks to learn continually by accumulating task information and reducing forgetting.

Contribution

The paper presents a novel synaptic intelligence approach that improves continual learning by efficiently storing task information and mitigating forgetting in neural networks.

Findings

Significantly reduces catastrophic forgetting in continual learning.

Maintains computational efficiency during learning.

Effective across multiple classification tasks.

Abstract

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.