Globally Optimal Training of Spiking Neural Networks via Parameter Reconstruction

TL;DR

This paper introduces a parameter reconstruction algorithm for training Spiking Neural Networks, addressing approximation errors from surrogate gradients and demonstrating advantages across multiple tasks.

Contribution

It extends convexification techniques to recurrent threshold networks and proposes a novel training method with improved scalability and robustness.

Findings

Consistent and significant performance improvements across tasks.

Demonstrates robustness to different model configurations.

Scalable to large-scale SNN training.

Abstract

Spiking Neural Networks (SNNs) have been proposed as biologically plausible and energy-efficient alternatives to conventional Artificial Neural Networks (ANNs). However, the training of SNN usually relies on surrogate gradients due to the non-differentiability of the spike function, introducing approximation errors that accumulate across layers. To address this challenge, we extend the work on convexification of parallel feedforward threshold networks to parallel recurrent threshold networks, which subsume parallel SNNs as a structured special case. Building on this theoretical framework, we propose a parameter reconstruction algorithm for SNN training that demonstrates consistent and significant advantages across various tasks, both as a standalone method and in combination with surrogate-gradient training. The ablations further demonstrate the data scalability and robustness to model configurations of our training algorithm, pointing toward its potential in large-scale SNN training.