DelGrad: Exact event-based gradients for training delays and weights on spiking neuromorphic hardware

TL;DR

DelGrad introduces an exact, event-based gradient computation method for training delays and weights in spiking neural networks, enhancing accuracy and efficiency on neuromorphic hardware by leveraging spike timing without extra variable tracking.

Contribution

It presents DelGrad, a novel analytical, event-based approach for exact gradient computation of delays and weights in SNNs, suitable for neuromorphic hardware.

Findings

Demonstrates memory efficiency and accuracy benefits of delays on neuromorphic hardware.

Shows delays can stabilize networks against noise.

Validates DelGrad on BrainScaleS-2 platform with chip-in-the-loop training.

Abstract

Spiking neural networks (SNNs) inherently rely on the timing of signals for representing and processing information. Incorporating trainable transmission delays, alongside synaptic weights, is crucial for shaping these temporal dynamics. While recent methods have shown the benefits of training delays and weights in terms of accuracy and memory efficiency, they rely on discrete time, approximate gradients, and full access to internal variables like membrane potentials. This limits their precision, efficiency, and suitability for neuromorphic hardware due to increased memory requirements and I/O bandwidth demands. To address these challenges, we propose DelGrad, an analytical, event-based method to compute exact loss gradients for both synaptic weights and delays. The inclusion of delays in the training process emerges naturally within our proposed formalism, enriching the model's search space with a temporal dimension. Moreover, DelGrad, grounded purely in spike timing, eliminates the need to track additional variables such as membrane potentials. To showcase this key advantage, we demonstrate the functionality and benefits of DelGrad on the BrainScaleS-2 neuromorphic platform, by training SNNs in a chip-in-the-loop fashion. For the first time, we experimentally demonstrate the memory efficiency and accuracy benefits of adding delays to SNNs on noisy mixed-signal hardware. Additionally, these experiments also reveal the potential of delays for stabilizing networks against noise. DelGrad opens a new way for training SNNs with delays on neuromorphic hardware, which results in fewer required parameters, higher accuracy and ease of hardware training.