Smooth Exact Gradient Descent Learning in Spiking Neural Networks

TL;DR

This paper introduces an exact gradient descent method for spiking neural networks that handles spike appearance and disappearance, enabling effective training of complex recurrent and deep networks.

Contribution

It presents a novel gradient descent approach that accounts for spike dynamics, allowing precise training of spiking neural networks with spike addition and removal.

Findings

Effective training of recurrent and deep spiking networks.

Gradient-based spike addition and removal demonstrated.

Applicable to various tasks and network setups.

Abstract

Gradient descent prevails in artificial neural network training, but seems inept for spiking neural networks as small parameter changes can cause sudden, disruptive (dis-)appearances of spikes. Here, we demonstrate exact gradient descent based on continuously changing spiking dynamics. These are generated by neuron models whose spikes vanish and appear at the end of a trial, where it cannot influence subsequent dynamics. This also enables gradient-based spike addition and removal. We illustrate our scheme with various tasks and setups, including recurrent and deep, initially silent networks.