Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

TL;DR

This paper introduces LSFormer, a novel Transformer-based Spiking Neural Network that uses local structure-aware self-attention and spiking response pooling to improve efficiency and accuracy in vision tasks.

Contribution

It proposes LSFormer, incorporating local dilated window mechanisms and new pooling/self-attention modules, addressing limitations of existing Transformer-based SNNs.

Findings

Achieves state-of-the-art performance on Tiny-ImageNet and N-CALTECH101 datasets.

Outperforms existing Transformer-based SNNs by 4.3% and 8.6% in top-1 accuracy.

Demonstrates potential for energy-efficient large-scale vision applications.

Abstract

Transformer-based Spiking Neural Networks (SNNs) integrate SNNs with global self-attention and have demonstrated impressive performance. However, existing Transformer-based SNNs suffer from two fundamental limitations. First, they typically employ max pooling layers to reduce the size of feature maps, but the max pooling captures only the strongest response and fails to comprehensively preserve representative regional features. Second, the global self-attention involves all global feature interactions, resulting in computational redundancy and quadratic computational complexity, thus conflicting with the sparse and energy-efficient characteristics of SNNs. To address these challenges, we develop Local Structure-Aware Spiking Transformer (LSFormer), a novel Transformer-based Spiking Neural Network that incorporates Spiking Response Pooling (SPooling) and Local Structure-Aware Spiking Self-Attention (LS-SSA). For the first time, our LSFormer leverages a local dilated window mechanism to capture both local details and long-range dependencies. Experimental results demonstrate that our LSFormer achieves state-of-the-art performance compared to existing advanced Transformer-based SNNs. Notably, on the more challenging static dataset Tiny-ImageNet and neuromorphic dataset N-CALTECH101, LSFormer substantially outperforms state-of-the-art baselines by 4.3\% and 8.6\% in top-1 classification accuracy, respectively. These results highlight the potential of LSFormer to advance energy-efficient spiking models toward practical deployment in large-scale vision applications.