Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

TL;DR

This paper introduces CPG-PE, a novel positional encoding method inspired by central pattern generators, significantly improving the performance of spiking neural networks on sequential tasks across multiple domains.

Contribution

The paper proposes CPG-PE, a biologically inspired positional encoding technique for SNNs, addressing the challenge of sequence modeling and demonstrating superior performance.

Findings

CPG-PE outperforms traditional sinusoidal PE in various tasks.

Mathematically, sinusoidal PE is a specific case of CPG dynamics.

Analysis reveals how SNNs encode positional information using CPG-PE.

Abstract

Spiking neural networks (SNNs) represent a promising approach to developing artificial neural networks that are both energy-efficient and biologically plausible. However, applying SNNs to sequential tasks, such as text classification and time-series forecasting, has been hindered by the challenge of creating an effective and hardware-friendly spike-form positional encoding (PE) strategy. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, we propose a novel PE technique for SNNs, termed CPG-PE. We demonstrate that the commonly used sinusoidal PE is mathematically a specific solution to the membrane potential dynamics of a particular CPG. Moreover, extensive experiments across various domains, including time-series forecasting, natural language processing, and image classification, show that SNNs with CPG-PE outperform their conventional counterparts. Additionally, we perform analysis experiments to elucidate the mechanism through which SNNs encode positional information and to explore the function of CPGs in the human brain. This investigation may offer valuable insights into the fundamental principles of neural computation. Our code is available at https://github.com/microsoft/SeqSNN.