Towards High-performance Spiking Transformers from ANN to SNN Conversion

TL;DR

This paper introduces a novel conversion method for transforming Transformer neural networks into spiking neural networks, achieving high accuracy with low latency and power consumption, thus advancing energy-efficient AI models.

Contribution

It presents the first successful ANN to SNN conversion for Spiking Transformers, incorporating an Expectation Compensation Module and Multi-Threshold Neuron to enhance performance.

Findings

Achieves 88.60% top-1 accuracy with 4 time steps

Reduces power consumption to 35% of original Transformer

Maintains high accuracy with only 1% loss

Abstract

Spiking neural networks (SNNs) show great potential due to their energy efficiency, fast processing capabilities, and robustness. There are two main approaches to constructing SNNs. Direct training methods require much memory, while conversion methods offer a simpler and more efficient option. However, current conversion methods mainly focus on converting convolutional neural networks (CNNs) to SNNs. Converting Transformers to SNN is challenging because of the presence of non-linear modules. In this paper, we propose an Expectation Compensation Module to preserve the accuracy of the conversion. The core idea is to use information from the previous T time-steps to calculate the expected output at time-step T. We also propose a Multi-Threshold Neuron and the corresponding Parallel Parameter normalization to address the challenge of large time steps needed for high accuracy, aiming to reduce network latency and power consumption. Our experimental results demonstrate that our approach achieves state-of-the-art performance. For example, we achieve a top-1 accuracy of 88.60\% with only a 1\% loss in accuracy using 4 time steps while consuming only 35\% of the original power of the Transformer. To our knowledge, this is the first successful Artificial Neural Network (ANN) to SNN conversion for Spiking Transformers that achieves high accuracy, low latency, and low power consumption on complex datasets. The source codes of the proposed method are available at https://github.com/h-z-h-cell/Transformer-to-SNN-ECMT.