Accelerating SNN Training with Stochastic Parallelizable Spiking Neurons

TL;DR

This paper introduces SPSN, a novel spiking neuron model that enables parallel processing over time, significantly accelerating SNN training while maintaining competitive performance.

Contribution

The paper proposes SPSN, a new neuron model that separates linear and non-linear components, allowing parallel computation and faster training of SNNs.

Findings

SPSN achieves comparable accuracy to state-of-the-art on SHD dataset.

Training speed is increased by a factor of 10 compared to LIF neurons.

For 10,000 time-step sequences, training is 4000 times faster.

Abstract

Spiking neural networks (SNN) are able to learn spatiotemporal features while using less energy, especially on neuromorphic hardware. The most widely used spiking neuron in deep learning is the Leaky Integrate and Fire (LIF) neuron. LIF neurons operate sequentially, however, since the computation of state at time t relies on the state at time t-1 being computed. This limitation is shared with Recurrent Neural Networks (RNN) and results in slow training on Graphics Processing Units (GPU). In this paper, we propose the Stochastic Parallelizable Spiking Neuron (SPSN) to overcome the sequential training limitation of LIF neurons. By separating the linear integration component from the non-linear spiking function, SPSN can be run in parallel over time. The proposed approach results in performance comparable with the state-of-the-art for feedforward neural networks on the Spiking Heidelberg Digits (SHD) dataset, outperforming LIF networks while training 10 times faster and outperforming non-spiking networks with the same network architecture. For longer input sequences of 10000 time-steps, we show that the proposed approach results in 4000 times faster training, thus demonstrating the potential of the proposed approach to accelerate SNN training for very large datasets.