Fluctuation-driven initialization for spiking neural network training

TL;DR

This paper introduces a fluctuation-driven initialization method for spiking neural networks that enhances training performance across various architectures and datasets by promoting fluctuation-driven firing in neurons.

Contribution

The authors propose a novel data-dependent weight initialization strategy inspired by brain activity, improving end-to-end training of SNNs with surrogate gradients.

Findings

SNNs initialized with the proposed method outperform standard initializations.

The strategy is effective across multiple SNN architectures and datasets.

It facilitates fluctuation-driven firing, leading to better learning performance.

Abstract

Spiking neural networks (SNNs) underlie low-power, fault-tolerant information processing in the brain and could constitute a power-efficient alternative to conventional deep neural networks when implemented on suitable neuromorphic hardware accelerators. However, instantiating SNNs that solve complex computational tasks in-silico remains a significant challenge. Surrogate gradient (SG) techniques have emerged as a standard solution for training SNNs end-to-end. Still, their success depends on synaptic weight initialization, similar to conventional artificial neural networks (ANNs). Yet, unlike in the case of ANNs, it remains elusive what constitutes a good initial state for an SNN. Here, we develop a general initialization strategy for SNNs inspired by the fluctuation-driven regime commonly observed in the brain. Specifically, we derive practical solutions for data-dependent weight initialization that ensure fluctuation-driven firing in the widely used leaky integrate-and-fire (LIF) neurons. We empirically show that SNNs initialized following our strategy exhibit superior learning performance when trained with SGs. These findings generalize across several datasets and SNN architectures, including fully connected, deep convolutional, recurrent, and more biologically plausible SNNs obeying Dale's law. Thus fluctuation-driven initialization provides a practical, versatile, and easy-to-implement strategy for improving SNN training performance on diverse tasks in neuromorphic engineering and computational neuroscience.