Advancing Deep Residual Learning by Solving the Crux of Degradation in Spiking Neural Networks

TL;DR

This paper introduces a novel residual block for deep spiking neural networks, enabling training of significantly deeper models with high accuracy and energy efficiency, advancing neuromorphic computing applications.

Contribution

The paper proposes a new residual block that allows training of ultra-deep SNNs, overcoming degradation issues and achieving state-of-the-art results on ImageNet.

Findings

Deep SNNs up to 482 layers trained without degradation

Achieved 76.02% accuracy on ImageNet with SNNs

Networks require only one spike per neuron on average for classification

Abstract

Despite the rapid progress of neuromorphic computing, the inadequate depth and the resulting insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. This negligence leads to impeded information flow and the accompanying degradation problem. In this paper, we identify the crux and then propose a novel residual block for SNNs, which is able to significantly extend the depth of directly trained SNNs, e.g., up to 482 layers on CIFAR-10 and 104 layers on ImageNet, without observing any slight degradation problem. We validate the effectiveness of our methods on both frame-based and neuromorphic datasets, and our SRM-ResNet104 achieves a superior result of 76.02% accuracy on ImageNet, the first time in the domain of directly trained SNNs. The great energy efficiency is estimated and the resulting networks need on average only one spike per neuron for classifying an input sample. We believe our powerful and scalable modeling will provide a strong support for further exploration of SNNs.