Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding

TL;DR

This paper introduces the CE-LIF spiking neuron model that enhances long-term sequential modeling in SNNs by integrating contextual embedding, leading to improved accuracy, convergence, and memory capacity.

Contribution

The paper proposes a novel CE-LIF neuron model with contextual embedding, enabling better long-term temporal modeling in SNNs, supported by theoretical analysis.

Findings

Superior classification accuracy on sequential tasks

Faster network convergence compared to existing models

Enhanced memory capacity in SNNs

Abstract

The human brain exhibits remarkable abilities in integrating temporally distant sensory inputs for decision-making. However, existing brain-inspired spiking neural networks (SNNs) have struggled to match their biological counterpart in modeling long-term temporal relationships. To address this problem, this paper presents a novel Contextual Embedding Leaky Integrate-and-Fire (CE-LIF) spiking neuron model. Specifically, the CE-LIF model incorporates a meticulously designed contextual embedding component into the adaptive neuronal firing threshold, thereby enhancing the memory storage of spiking neurons and facilitating effective sequential modeling. Additionally, theoretical analysis is provided to elucidate how the CE-LIF model enables long-term temporal credit assignment. Remarkably, when compared to state-of-the-art recurrent SNNs, feedforward SNNs comprising the proposed CE-LIF neurons demonstrate superior performance across extensive sequential modeling tasks in terms of classification accuracy, network convergence speed, and memory capacity.