Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation

TL;DR

Spike2Former introduces an efficient spiking transformer architecture that significantly improves image segmentation performance and stability in SNNs, setting new state-of-the-art results on multiple datasets.

Contribution

The paper proposes Spike2Former with normalized integer spiking neurons, addressing stability and performance issues in SNNs for segmentation tasks.

Findings

Achieved +12.7% mIoU on ADE20K

Improved efficiency by up to 6.6x

Set new state-of-the-art results in SNN segmentation

Abstract

Spiking Neural Networks (SNNs) have a low-power advantage but perform poorly in image segmentation tasks. The reason is that directly converting neural networks with complex architectural designs for segmentation tasks into spiking versions leads to performance degradation and non-convergence. To address this challenge, we first identify the modules in the architecture design that lead to the severe reduction in spike firing, make targeted improvements, and propose Spike2Former architecture. Second, we propose normalized integer spiking neurons to solve the training stability problem of SNNs with complex architectures. We set a new state-of-the-art for SNNs in various semantic segmentation datasets, with a significant improvement of +12.7% mIoU and 5.0 efficiency on ADE20K, +14.3% mIoU and 5.2 efficiency on VOC2012, and +9.1% mIoU and 6.6 efficiency on CityScapes.