DS2TA: Denoising Spiking Transformer with Attenuated Spatiotemporal Attention

TL;DR

DS2TA introduces a novel denoising spiking transformer with attenuated spatiotemporal attention, significantly improving robustness and accuracy in vision tasks on neuromorphic hardware without extra parameters.

Contribution

It proposes a new spiking attenuated spatiotemporal attention mechanism that leverages input firing correlations in space and time, enhancing spiking neural network performance.

Findings

Achieves 94.92% top-1 accuracy on CIFAR10

Attains 77.47% top-1 accuracy on CIFAR100

Performs well on neuromorphic datasets like CIFAR10-DVS and DVS-Gesture

Abstract

Vision Transformers (ViT) are current high-performance models of choice for various vision applications. Recent developments have given rise to biologically inspired spiking transformers that thrive in ultra-low power operations on neuromorphic hardware, however, without fully unlocking the potential of spiking neural networks. We introduce DS2TA, a Denoising Spiking transformer with attenuated SpatioTemporal Attention, designed specifically for vision applications. DS2TA introduces a new spiking attenuated spatiotemporal attention mechanism that considers input firing correlations occurring in both time and space, thereby fully harnessing the computational power of spiking neurons at the core of the transformer architecture. Importantly, DS2TA facilitates parameter-efficient spatiotemporal attention computation without introducing extra weights. DS2TA employs efficient hashmap-based nonlinear spiking attention denoisers to enhance the robustness and expressive power of spiking attention maps. DS2TA demonstrates state-of-the-art performances on several widely adopted static image and dynamic neuromorphic datasets. Operated over 4 time steps, DS2TA achieves 94.92% top-1 accuracy on CIFAR10 and 77.47% top-1 accuracy on CIFAR100, as well as 79.1% and 94.44% on CIFAR10-DVS and DVS-Gesture using 10 time steps.