MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion

TL;DR

MSVIT introduces a multi-scale spiking attention mechanism to improve feature extraction in SNN-based vision transformers, achieving state-of-the-art performance across multiple datasets.

Contribution

The paper proposes MSVIT, a novel spiking transformer architecture with multi-scale attention, addressing feature extraction bottlenecks in existing SNN-transformer models.

Findings

MSVIT outperforms existing SNN-based models on main datasets.

Multi-scale spiking attention enhances feature extraction.

Achieves state-of-the-art results among SNN-transformer architectures.

Abstract

The combination of Spiking Neural Networks (SNNs) with Vision Transformer architectures has garnered significant attention due to their potential for energy-efficient and high-performance computing paradigms. However, a substantial performance gap still exists between SNN-based and ANN-based transformer architectures. While existing methods propose spiking self-attention mechanisms that are successfully combined with SNNs, the overall architectures proposed by these methods suffer from a bottleneck in effectively extracting features from different image scales. In this paper, we address this issue and propose MSVIT. This novel spike-driven Transformer architecture firstly uses multi-scale spiking attention (MSSA) to enhance the capabilities of spiking attention blocks. We validate our approach across various main datasets. The experimental results show that MSVIT outperforms existing SNN-based models, positioning itself as a state-of-the-art solution among SNN-transformer architectures. The codes are available at https://github.com/Nanhu-AI-Lab/MSViT.