STF: Shallow-Level Temporal Feedback to Enhance Spiking Transformers

TL;DR

This paper introduces STF, a lightweight module that improves the performance of Transformer-based Spiking Neural Networks by incorporating shallow-level temporal feedback, enhancing spike pattern diversity and robustness.

Contribution

The paper proposes STF, a novel shallow-level temporal feedback module with TSPE and TF, which boosts SNN performance with lower overhead compared to deep feedback designs.

Findings

Consistent performance improvements on CIFAR-10, CIFAR-100, and ImageNet-1K datasets.

Enhanced spike pattern diversity correlates with better accuracy.

Outperforms existing coding schemes in robustness and temporal sensitivity.

Abstract

Transformer-based Spiking Neural Networks (SNNs) suffer from a great performance gap compared to floating-point \mbox{Artificial} Neural Networks (ANNs) due to the binary nature of spike trains. Recent efforts have introduced deep-level feedback loops to transmit high-level semantic information to narrow this gap. However, these designs often span \mbox{multiple} deep layers, resulting in costly feature transformations, higher parameter overhead, increased energy consumption, and longer inference latency. To address this issue, we propose Shallow-level Temporal Feedback (STF), a lightweight plug-and-play module for the encoding layer, which consists of Temporal-Spatial Position Embedding (TSPE) and Temporal Feedback (TF). Extensive experiments show that STF consistently improves performance across various Transformer-based SNN backbones on static datasets, including CIFAR-10, CIFAR-100, and ImageNet-1K, under different spike timestep settings. Further analysis reveals that STF enhances the diversity of spike patterns, which is key to performance gain. Moreover, evaluations on adversarial robustness and temporal sensitivity confirm that STF outperforms direct coding and its variants, highlighting its potential as a new spike encoding scheme for static scenarios. Our code will be released upon acceptance.