Spikingformer: A Key Foundation Model for Spiking Neural Networks

TL;DR

Spikingformer introduces a biologically plausible spiking transformer backbone that reduces non-spike computations in SNNs, enhancing energy efficiency and versatility across diverse tasks.

Contribution

The paper proposes Spikingformer, a novel SNN backbone combining residual connections and self-attention to eliminate non-spike computations while maintaining global modeling.

Findings

Effective across 13 datasets including images, neuromorphic data, and language.

Outperforms existing SNN backbones in energy efficiency and accuracy.

Establishes a new benchmark for spiking neural network performance.

Abstract

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks, due to their event-driven spiking computation. However, some foundation SNN backbones (including Spikformer and SEW ResNet) suffer from non-spike computations (integer-float multiplications) caused by the structure of their residual connections. These non-spike computations increase SNNs' power consumption and make them unsuitable for deployment on mainstream neuromorphic hardware. In this paper, we analyze the spike-driven behavior of the residual connection methods in SNNs. We then present Spikingformer, a novel spiking transformer backbone that merges the MS Residual connection with Self-Attention in a biologically plausible way to address the non-spike computation challenge in Spikformer while maintaining global modeling capabilities. We evaluate Spikingformer across 13 datasets spanning large static images, neuromorphic data, and natural language tasks, and demonstrate the effectiveness and universality of Spikingformer, setting a vital benchmark for spiking neural networks. In addition, with the spike-driven features and global modeling capabilities, Spikingformer is expected to become a more efficient general-purpose SNN backbone towards energy-efficient artificial intelligence. Code: https://github.com/TheBrainLab/Spikingformer