Investigating Continuous Learning in Spiking Neural Networks

TL;DR

This study explores the potential of spiking neural networks for continuous learning, showing they can partially mitigate catastrophic forgetting compared to conventional models, but further research is needed.

Contribution

It compares conventional models and spiking neural networks in continuous learning, highlighting the potential of SNNs to reduce catastrophic forgetting.

Findings

SNN models retain more information about previous classes.

Conventional models experience sharp performance drops due to forgetting.

SNN output probabilities indicate better retention of past knowledge.

Abstract

In this paper, the use of third-generation machine learning, also known as spiking neural network architecture, for continuous learning was investigated and compared to conventional models. The experimentation was divided into three separate phases. The first phase focused on training the conventional models via transfer learning. The second phase trains a Nengo model from their library. Lastly, each conventional model is converted into a spiking neural network and trained. Initial results from phase 1 are inline with known knowledge about continuous learning within current machine learning literature. All models were able to correctly identify the current classes, but they would immediately see a sharp performance drop in previous classes due to catastrophic forgetting. However, the SNN models were able to retain some information about previous classes. Although many of the previous classes were still identified as the current trained classes, the output probabilities showed a higher than normal value to the actual class. This indicates that the SNN models do have potential to overcome catastrophic forgetting but much work is still needed.