DISTA: Denoising Spiking Transformer with intrinsic plasticity and spatiotemporal attention

TL;DR

DISTA introduces a novel denoising spiking transformer that leverages intrinsic and spatiotemporal attention mechanisms, achieving state-of-the-art results in vision tasks with ultra-low latency and power efficiency.

Contribution

The paper proposes DISTA, a spiking transformer with intrinsic plasticity and spatiotemporal attention, enhancing neural computation and performance in vision applications.

Findings

Achieves 96.26% top-1 accuracy on CIFAR10 with 6 time steps.

Outperforms previous spiking transformers on neuromorphic datasets.

Uses joint training of synaptic and intrinsic plasticity for optimal performance.

Abstract

Among the array of neural network architectures, the Vision Transformer (ViT) stands out as a prominent choice, acclaimed for its exceptional expressiveness and consistent high performance in various vision applications. Recently, the emerging Spiking ViT approach has endeavored to harness spiking neurons, paving the way for a more brain-inspired transformer architecture that thrives in ultra-low power operations on dedicated neuromorphic hardware. Nevertheless, this approach remains confined to spatial self-attention and doesn't fully unlock the potential of spiking neural networks. We introduce DISTA, a Denoising Spiking Transformer with Intrinsic Plasticity and SpatioTemporal Attention, designed to maximize the spatiotemporal computational prowess of spiking neurons, particularly for vision applications. DISTA explores two types of spatiotemporal attentions: intrinsic neuron-level attention and network-level attention with explicit memory. Additionally, DISTA incorporates an efficient nonlinear denoising mechanism to quell the noise inherent in computed spatiotemporal attention maps, thereby resulting in further performance gains. Our DISTA transformer undergoes joint training involving synaptic plasticity (i.e., weight tuning) and intrinsic plasticity (i.e., membrane time constant tuning) and delivers state-of-the-art performances across several static image and dynamic neuromorphic datasets. With only 6 time steps, DISTA achieves remarkable top-1 accuracy on CIFAR10 (96.26%) and CIFAR100 (79.15%), as well as 79.1% on CIFAR10-DVS using 10 time steps.