BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning

TL;DR

BSViT introduces a novel burst spiking vision transformer with dual-channel self-attention and local masking, significantly improving accuracy and efficiency in visual learning tasks.

Contribution

It proposes a dual-channel burst spiking self-attention mechanism and patch adjacency masking to enhance representational capacity and reduce computation in spiking vision transformers.

Findings

BSViT outperforms existing spiking Transformers on static and event-based benchmarks.

The model maintains energy efficiency through addition-only operations.

Incorporating burst spike coding increases spike-level representational capacity.

Abstract

Spiking Vision Transformers (S-ViTs) offer a promising framework for energy-efficient visual learning. However, existing designs remain limited by two fundamental issues: the restricted information capacity of binary spike coding and the dense token interactions introduced by global self-attention. To address these challenges, this work proposes BSViT, a burst spiking-driven Vision Transformer featuring a Dual-Channel Burst Spiking Self-Attention (DBSSA) mechanism. DBSSA encodes queries with binary spikes and keys with burst spikes to enhance representational capacity. The value pathway adopts dual excitatory and inhibitory binary channels, enabling signed modulation and richer spike interactions. Importantly, the entire attention operation preserves addition-only computation, ensuring compatibility with energy-efficient neuromorphic hardware. To further reduce spike activity and incorporate spatial priors, a patch adjacency masking strategy is introduced to restrict attention to local neighborhoods, resulting in structure-aware sparsity and reduced computational overhead. In addition, burst spike coding is systematically integrated across the network to increase spike-level representational capacity beyond conventional binary spiking. Extensive experiments on both static and event-based vision benchmarks demonstrate that BSViT consistently outperforms existing spiking Transformers in accuracy while maintaining competitive energy efficiency.