Towards Understanding the Effect of Leak in Spiking Neural Networks

TL;DR

This paper investigates the impact of leak in spiking neural networks, showing that leaky models enhance robustness and generalization by filtering high-frequency noise, despite reducing computational sparsity.

Contribution

It provides a comparative analysis of leaky versus non-leaky neuron models, highlighting the benefits of leak in robustness and noise filtering in SNNs.

Findings

Leaky neuron models improve robustness and generalization.

Leak reduces high-frequency noise in inputs.

Leak decreases computational sparsity.

Abstract

Spiking Neural Networks (SNNs) are being explored to emulate the astounding capabilities of human brain that can learn and compute functions robustly and efficiently with noisy spiking activities. A variety of spiking neuron models have been proposed to resemble biological neuronal functionalities. With varying levels of bio-fidelity, these models often contain a leak path in their internal states, called membrane potentials. While the leaky models have been argued as more bioplausible, a comparative analysis between models with and without leak from a purely computational point of view demands attention. In this paper, we investigate the questions regarding the justification of leak and the pros and cons of using leaky behavior. Our experimental results reveal that leaky neuron model provides improved robustness and better generalization compared to models with no leak. However, leak decreases the sparsity of computation contrary to the common notion. Through a frequency domain analysis, we demonstrate the effect of leak in eliminating the high-frequency components from the input, thus enabling SNNs to be more robust against noisy spike-inputs.