Linearized Bregman Iterations for Sparse Spiking Neural Networks

TL;DR

This paper introduces Linearized Bregman Iterations as a novel optimizer for training sparse Spiking Neural Networks, significantly reducing parameters while maintaining accuracy, thus enhancing energy efficiency in neuromorphic computing.

Contribution

It proposes a new optimization method, LBI, combined with AdaBreg, to enforce sparsity in SNNs, improving efficiency without sacrificing performance.

Findings

LBI reduces active parameters by about 50%.

Models maintain accuracy comparable to Adam-trained models.

Demonstrates potential for efficient neuromorphic learning.

Abstract

Spiking Neural Networks (SNNs) offer an energy efficient alternative to conventional Artificial Neural Networks (ANNs) but typically still require a large number of parameters. This work introduces Linearized Bregman Iterations (LBI) as an optimizer for training SNNs, enforcing sparsity through iterative minimization of the Bregman distance and proximal soft thresholding updates. To improve convergence and generalization, we employ the AdaBreg optimizer, a momentum and bias corrected Bregman variant of Adam. Experiments on three established neuromorphic benchmarks, i.e. the Spiking Heidelberg Digits (SHD), the Spiking Speech Commands (SSC), and the Permuted Sequential MNIST (PSMNIST) datasets, show that LBI based optimization reduces the number of active parameters by about 50% while maintaining accuracy comparable to models trained with the Adam optimizer, demonstrating the potential of convex sparsity inducing methods for efficient neuromorphic learning.