Training Deep Spiking Neural Networks

TL;DR

This paper demonstrates direct training of deep spiking neural networks using surrogate gradient backpropagation, addressing training challenges and achieving efficient inference with fewer time steps compared to converted models.

Contribution

It introduces methods to overcome gradient issues in deep SNNs and applies batch normalization, enabling training of ResNet50 on CIFAR100 and Imagenette datasets.

Findings

Deep SNNs can be trained directly with surrogate gradients.

Training SNNs with fewer inference steps is feasible.

Accuracy is slightly lower than ANNs but with significant efficiency gains.

Abstract

Computation using brain-inspired spiking neural networks (SNNs) with neuromorphic hardware may offer orders of magnitude higher energy efficiency compared to the current analog neural networks (ANNs). Unfortunately, training SNNs with the same number of layers as state of the art ANNs remains a challenge. To our knowledge the only method which is successful in this regard is supervised training of ANN and then converting it to SNN. In this work we directly train deep SNNs using backpropagation with surrogate gradient and find that due to implicitly recurrent nature of feed forward SNN's the exploding or vanishing gradient problem severely hinders their training. We show that this problem can be solved by tuning the surrogate gradient function. We also propose using batch normalization from ANN literature on input currents of SNN neurons. Using these improvements we show that is is possible to train SNN with ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets. The trained SNN falls behind in accuracy compared to analogous ANN but requires several orders of magnitude less inference time steps (as low as 10) to reach good accuracy compared to SNNs obtained by conversion from ANN which require on the order of 1000 time steps.