Spiking Transformer with Spatial-Temporal Attention

TL;DR

This paper introduces STAtten, a novel spike-based transformer architecture that effectively captures both spatial and temporal dependencies, significantly enhancing performance on various static and neuromorphic datasets while maintaining computational efficiency.

Contribution

STAtten is a simple, efficient architecture that integrates spatial-temporal attention into spike-based transformers without requiring major redesigns.

Findings

Improves accuracy on CIFAR10/100, ImageNet, CIFAR10-DVS, and N-Caltech101 datasets.

Maintains computational complexity comparable to spatial-only models.

Demonstrates significant performance gains through extensive experiments.

Abstract

Spike-based Transformer presents a compelling and energy-efficient alternative to traditional Artificial Neural Network (ANN)-based Transformers, achieving impressive results through sparse binary computations. However, existing spike-based transformers predominantly focus on spatial attention while neglecting crucial temporal dependencies inherent in spike-based processing, leading to suboptimal feature representation and limited performance. To address this limitation, we propose Spiking Transformer with Spatial-Temporal Attention (STAtten), a simple and straightforward architecture that efficiently integrates both spatial and temporal information in the self-attention mechanism. STAtten introduces a block-wise computation strategy that processes information in spatial-temporal chunks, enabling comprehensive feature capture while maintaining the same computational complexity as previous spatial-only approaches. Our method can be seamlessly integrated into existing spike-based transformers without architectural overhaul. Extensive experiments demonstrate that STAtten significantly improves the performance of existing spike-based transformers across both static and neuromorphic datasets, including CIFAR10/100, ImageNet, CIFAR10-DVS, and N-Caltech101. The code is available at https://github.com/Intelligent-Computing-Lab-Yale/STAtten