Deep Residual Learning in Spiking Neural Networks

TL;DR

This paper introduces the spike-element-wise (SEW) ResNet, a novel residual learning architecture for deep Spiking Neural Networks that overcomes previous training limitations, enabling the training of networks with over 100 layers and achieving superior accuracy.

Contribution

The paper proposes the SEW ResNet architecture for deep SNNs, demonstrating its ability to implement residual learning and train very deep networks effectively.

Findings

SEW ResNet outperforms state-of-the-art SNNs in accuracy and efficiency.

Enables training of SNNs with more than 100 layers.

Achieves higher performance by simply adding more layers.

Abstract

Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.