Energy Efficient Training of SNN using Local Zeroth Order Method

TL;DR

This paper introduces a local zeroth-order method for training spiking neural networks that reduces energy consumption and computational cost, achieving significant speed-ups while maintaining accuracy.

Contribution

It proposes a novel zeroth-order training technique for SNNs that connects with existing surrogate methods and enhances energy efficiency on GPUs.

Findings

Requires less than 1% neurons active during backpropagation

Achieves 100x speed-up in backward computation

Maintains comparable accuracy to state-of-the-art methods

Abstract

Spiking neural networks are becoming increasingly popular for their low energy requirement in real-world tasks with accuracy comparable to the traditional ANNs. SNN training algorithms face the loss of gradient information and non-differentiability due to the Heaviside function in minimizing the model loss over model parameters. To circumvent the problem surrogate method uses a differentiable approximation of the Heaviside in the backward pass, while the forward pass uses the Heaviside as the spiking function. We propose to use the zeroth order technique at the neuron level to resolve this dichotomy and use it within the automatic differentiation tool. As a result, we establish a theoretical connection between the proposed local zeroth-order technique and the existing surrogate methods and vice-versa. The proposed method naturally lends itself to energy-efficient training of SNNs on GPUs. Experimental results with neuromorphic datasets show that such implementation requires less than 1 percent neurons to be active in the backward pass, resulting in a 100x speed-up in the backward computation time. Our method offers better generalization compared to the state-of-the-art energy-efficient technique while maintaining similar efficiency.