Improving Surrogate Gradient Learning in Spiking Neural Networks via Regularization and Normalization

TL;DR

This paper explores regularization and normalization techniques to enhance surrogate gradient learning in spiking neural networks, aiming to improve their accuracy and efficiency for potential neuromorphic hardware deployment.

Contribution

It introduces novel regularization and normalization methods specifically designed to improve surrogate gradient training in SNNs, addressing their accuracy limitations.

Findings

Improved training stability of SNNs

Enhanced accuracy over baseline models

Effective regularization techniques for SNNs

Abstract

Spiking neural networks (SNNs) are different from the classical networks used in deep learning: the neurons communicate using electrical impulses called spikes, just like biological neurons. SNNs are appealing for AI technology, because they could be implemented on low power neuromorphic chips. However, SNNs generally remain less accurate than their analog counterparts. In this report, we examine various regularization and normalization techniques with the goal of improving surrogate gradient learning in SNNs.