Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer

TL;DR

This paper presents a training-free method for converting Artificial Neural Networks to Spiking Neural Networks in Transformer models, achieving high accuracy and low latency without retraining, thus enabling energy-efficient deployment.

Contribution

The authors introduce a novel Multi-basis Exponential Decay neuron that effectively approximates nonlinear operations in Transformers without weight modifications, simplifying ANN-to-SNN conversion.

Findings

Achieves near-lossless conversion accuracy across multiple tasks and architectures.

Reduces latency significantly compared to existing methods.

Eliminates the need for fine-tuning pretrained ANNs.

Abstract

Leveraging the event-driven paradigm, Spiking Neural Networks (SNNs) offer a promising approach for energy-efficient Transformer architectures.While ANN-to-SNN conversion avoids the high training cost of directly trained Spiking Transformers, existing approaches still struggle to handle the nonlinear operations within Transformer blocks, and often require additional fine-tuning of pretrained ANNs.To address these limitations, we propose a training-free and high-performance ANN-to-SNN conversion framework tailored for Transformer architectures. Specifically, we introduce a Multi-basis Exponential Decay (MBE) neuron that combines exponential decay with a multi-basis encoding strategy to effectively approximate nonlinear operations, eliminating the need for weight modifications in pretrained ANNs.Extensive experiments across diverse tasks (CV, NLU, NLG) and mainstream Transformer architectures (ViT, RoBERTa, GPT-2) demonstrate that our method achieves near-lossless conversion accuracy with significantly lower latency. This provides a promising pathway for the efficient and scalable deployment of Spiking Transformers in real-world applications.