Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation

TL;DR

This paper introduces a method to explicitly include spike counts in the loss function of spiking neural networks, enabling simultaneous optimization of accuracy and sparsity, leading to significant reductions in spiking activity.

Contribution

It proposes a novel training approach that incorporates spike sparsity directly into the loss function, improving energy efficiency without sacrificing accuracy.

Findings

Achieved up to 70.1% reduction in spiking activity at iso-accuracy.

Reduced spiking activity by 73.3% with only 1% accuracy loss.

Demonstrated effectiveness on the Cifar-10 dataset.

Abstract

Spiking Neural Networks (SNNs) are being explored for their potential energy efficiency resulting from sparse, event-driven computations. Many recent works have demonstrated effective backpropagation for deep Spiking Neural Networks (SNNs) by approximating gradients over discontinuous neuron spikes or firing events. A beneficial side-effect of these surrogate gradient spiking backpropagation algorithms is that the spikes, which trigger additional computations, may now themselves be directly considered in the gradient calculations. We propose an explicit inclusion of spike counts in the loss function, along with a traditional error loss, causing the backpropagation learning algorithms to optimize weight parameters for both accuracy and spiking sparsity. As supported by existing theory of over-parameterized neural networks, there are many solution states with effectively equivalent accuracy. As such, appropriate weighting of the two loss goals during training in this multi-objective optimization process can yield an improvement in spiking sparsity without a significant loss of accuracy. We additionally explore a simulated annealing-inspired loss weighting technique to increase the weighting for sparsity as training time increases. Our preliminary results on the Cifar-10 dataset show up to 70.1% reduction in spiking activity with iso-accuracy compared to an equivalent SNN trained only for accuracy and up to 73.3% reduction in spiking activity if allowed a trade-off of 1% reduction in classification accuracy.