Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks

TL;DR

This paper provides a theoretical foundation for surrogate gradient learning in spiking neural networks by relating it to stochastic automatic differentiation, supporting its practical effectiveness.

Contribution

It establishes a theoretical link between surrogate gradients and stochastic automatic differentiation in spiking neural networks, clarifying their validity and applicability.

Findings

Surrogate gradients are theoretically justified via stochastic automatic differentiation.

Empirical results confirm surrogate gradients' effectiveness in stochastic multi-layer networks.

Surrogate gradients are not generally derivatives of a surrogate loss, but are effective in practice.

Abstract

Training spiking neural networks to approximate universal functions is essential for studying information processing in the brain and for neuromorphic computing. Yet the binary nature of spikes poses a challenge for direct gradient-based training. Surrogate gradients have been empirically successful in circumventing this problem, but their theoretical foundation remains elusive. Here, we investigate the relation of surrogate gradients to two theoretically well-founded approaches. On the one hand, we consider smoothed probabilistic models, which, due to the lack of support for automatic differentiation, are impractical for training multi-layer spiking neural networks but provide derivatives equivalent to surrogate gradients for single neurons. On the other hand, we investigate stochastic automatic differentiation, which is compatible with discrete randomness but has not yet been used to train spiking neural networks. We find that the latter gives surrogate gradients a theoretical basis in stochastic spiking neural networks, where the surrogate derivative matches the derivative of the neuronal escape noise function. This finding supports the effectiveness of surrogate gradients in practice and suggests their suitability for stochastic spiking neural networks. However, surrogate gradients are generally not gradients of a surrogate loss despite their relation to stochastic automatic differentiation. Nevertheless, we empirically confirm the effectiveness of surrogate gradients in stochastic multi-layer spiking neural networks and discuss their relation to deterministic networks as a special case. Our work gives theoretical support to surrogate gradients and the choice of a suitable surrogate derivative in stochastic spiking neural networks.