Directly Training Temporal Spiking Neural Network with Sparse Surrogate Gradient

TL;DR

This paper introduces Masked Surrogate Gradients and a temporally weighted output method to improve the training and sparsity of Spiking Neural Networks, achieving state-of-the-art results.

Contribution

It proposes novel techniques, MSG and TWO, to enhance direct training of SNNs by balancing effectiveness and sparsity, leading to better generalization.

Findings

MSG improves training effectiveness while maintaining sparsity.

TWO enhances decoding accuracy by emphasizing important timesteps.

The combined approach surpasses current state-of-the-art methods.

Abstract

Brain-inspired Spiking Neural Networks (SNNs) have attracted much attention due to their event-based computing and energy-efficient features. However, the spiking all-or-none nature has prevented direct training of SNNs for various applications. The surrogate gradient (SG) algorithm has recently enabled spiking neural networks to shine in neuromorphic hardware. However, introducing surrogate gradients has caused SNNs to lose their original sparsity, thus leading to the potential performance loss. In this paper, we first analyze the current problem of direct training using SGs and then propose Masked Surrogate Gradients (MSGs) to balance the effectiveness of training and the sparseness of the gradient, thereby improving the generalization ability of SNNs. Moreover, we introduce a temporally weighted output (TWO) method to decode the network output, reinforcing the importance of correct timesteps. Extensive experiments on diverse network structures and datasets show that training with MSG and TWO surpasses the SOTA technique.