Synaptic metaplasticity in binarized neural networks

TL;DR

This paper introduces a biologically inspired metaplasticity mechanism for binarized neural networks to mitigate catastrophic forgetting, a common issue in deep learning models, by mimicking synaptic consolidation processes.

Contribution

The work adapts the concept of synaptic metaplasticity from neuroscience to improve the retention capabilities of binarized neural networks.

Findings

Metaplasticity reduces catastrophic forgetting in binarized neural networks.

Improved task retention demonstrated in experiments.

Biological plausibility enhances neural network robustness.

Abstract

Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one. Neuroscience suggests that biological synapses avoid this issue through the process of synaptic consolidation and metaplasticity: the plasticity itself changes upon repeated synaptic events. In this work, we show that this concept of metaplasticity can be transferred to a particular type of deep neural networks, binarized neural networks, to reduce catastrophic forgetting.