ADMM-Based Training for Spiking Neural Networks

TL;DR

This paper introduces an ADMM-based training algorithm for spiking neural networks, addressing limitations of surrogate gradient methods by providing a more accurate and scalable optimization approach.

Contribution

It presents the first formulation of SNN training as an ADMM-based iterative optimization with closed-form updates, improving training accuracy and scalability.

Findings

Demonstrates convergence of the ADMM-based optimizer

Shows potential for training deeper SNN architectures

Highlights advantages over surrogate gradient methods

Abstract

In recent years, spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption. However, they still lack a dedicated and efficient training algorithm. The popular backpropagation with surrogate gradients, adapted from stochastic gradient descent (SGD)-derived algorithms, has several drawbacks when used as an optimizer for SNNs. Specifically, the approximation introduced by the use of surrogate gradients leads to numerical imprecision, poor tracking of SNN firing times at training time, and, in turn, poor scalability. In this paper, we propose a novel SNN training method based on the alternating direction method of multipliers (ADMM). Our ADMM-based training aims to solve the problem of the SNN step function's non-differentiability by taking an entirely new approach with respect to gradient backpropagation. For the first time, we formulate the SNN training problem as an ADMM-based iterative optimization, derive closed-form updates, and empirically show the optimizer's convergence, its great potential, and discuss future and promising research directions to improve the method to different layer types and deeper architectures.