Temporal Surrogate Back-propagation for Spiking Neural Networks

TL;DR

This paper introduces a temporal surrogate back-propagation method for spiking neural networks, addressing the non-differentiability issue by incorporating the reset mechanism's temporal dependency, but finds limited practical benefit on larger datasets.

Contribution

It provides a theoretical analysis of the missing temporal term in surrogate gradient methods for SNNs and evaluates its impact empirically.

Findings

Adding the temporal dependency improves robustness to learning-rate changes on toy data.

The missing term offers limited benefits on larger datasets like CIFAR-10.

Ignoring the term is often justified due to computational overhead.

Abstract

Spiking neural networks (SNN) are usually more energy-efficient as compared to Artificial neural networks (ANN), and the way they work has a great similarity with our brain. Back-propagation (BP) has shown its strong power in training ANN in recent years. However, since spike behavior is non-differentiable, BP cannot be applied to SNN directly. Although prior works demonstrated several ways to approximate the BP-gradient in both spatial and temporal directions either through surrogate gradient or randomness, they omitted the temporal dependency introduced by the reset mechanism between each step. In this article, we target on theoretical completion and investigate the effect of the missing term thoroughly. By adding the temporal dependency of the reset mechanism, the new algorithm is more robust to learning-rate adjustments on a toy dataset but does not show much improvement on larger learning tasks like CIFAR-10. Empirically speaking, the benefits of the missing term are not worth the additional computational overhead. In many cases, the missing term can be ignored.