SNAP: Stopping Catastrophic Forgetting in Hebbian Learning with Sigmoidal Neuronal Adaptive Plasticity

TL;DR

This paper introduces SNAP, a sigmoidal weight growth mechanism inspired by biological neurons, which effectively prevents catastrophic forgetting in Hebbian Learning models by stabilizing weights after reaching certain strengths.

Contribution

The paper proposes a novel sigmoidal adaptive plasticity mechanism for Hebbian Learning that mitigates catastrophic forgetting, unlike traditional linear or exponential weight updates.

Findings

SNAP prevents forgetting in Hebbian Learning models.

SNAP does not prevent forgetting in SGD-based models.

Biological inspiration improves artificial neural network stability.

Abstract

Artificial Neural Networks (ANNs) suffer from catastrophic forgetting, where the learning of new tasks causes the catastrophic forgetting of old tasks. Existing Machine Learning (ML) algorithms, including those using Stochastic Gradient Descent (SGD) and Hebbian Learning typically update their weights linearly with experience i.e., independently of their current strength. This contrasts with biological neurons, which at intermediate strengths are very plastic, but consolidate with Long-Term Potentiation (LTP) once they reach a certain strength. We hypothesize this mechanism might help mitigate catastrophic forgetting. We introduce Sigmoidal Neuronal Adaptive Plasticity (SNAP) an artificial approximation to Long-Term Potentiation for ANNs by having the weights follow a sigmoidal growth behaviour allowing the weights to consolidate and stabilize when they reach sufficiently large or small values. We then compare SNAP to linear weight growth and exponential weight growth and see that SNAP completely prevents the forgetting of previous tasks for Hebbian Learning but not for SGD-base learning.