ILIF: Temporal Inhibitory Leaky Integrate-and-Fire Neuron for Overactivation in Spiking Neural Networks

TL;DR

This paper introduces the ILIF neuron model for spiking neural networks, addressing the gamma dilemma by balancing overactivation and gradient vanishing, leading to improved energy efficiency, training stability, and accuracy.

Contribution

The paper proposes a biologically inspired ILIF neuron model that mitigates overactivation in SNNs, enhancing training stability and energy efficiency compared to existing surrogate gradient methods.

Findings

ILIF reduces neuron firing rates and energy consumption.

ILIF stabilizes training and improves accuracy.

Theoretical analysis confirms ILIF's effectiveness in addressing the gamma dilemma.

Abstract

The Spiking Neural Network (SNN) has drawn increasing attention for its energy-efficient, event-driven processing and biological plausibility. To train SNNs via backpropagation, surrogate gradients are used to approximate the non-differentiable spike function, but they only maintain nonzero derivatives within a narrow range of membrane potentials near the firing threshold, referred to as the surrogate gradient support width gamma. We identify a major challenge, termed the dilemma of gamma: a relatively large gamma leads to overactivation, characterized by excessive neuron firing, which in turn increases energy consumption, whereas a small gamma causes vanishing gradients and weakens temporal dependencies. To address this, we propose a temporal Inhibitory Leaky Integrate-and-Fire (ILIF) neuron model, inspired by biological inhibitory mechanisms. This model incorporates interconnected inhibitory units for membrane potential and current, effectively mitigating overactivation while preserving gradient propagation. Theoretical analysis demonstrates ILIF effectiveness in overcoming the gamma dilemma, and extensive experiments on multiple datasets show that ILIF improves energy efficiency by reducing firing rates, stabilizes training, and enhances accuracy. The code is available at github.com/kaisun1/ILIF.