STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking

TL;DR

STEP is a comprehensive benchmarking platform for Spiking Transformers that enables fair comparison, reproducibility, and systematic analysis across various tasks, components, and models in the field.

Contribution

The paper introduces STEP, a unified, modular framework for evaluating Spiking Transformers, facilitating standardized benchmarking and in-depth analysis of design choices.

Findings

Current Spiking Transformers rely heavily on convolutional frontends.

Quantized ANNs can achieve comparable or better energy efficiency.

Spiking Transformers lack strong temporal modeling capabilities.

Abstract

Spiking Transformers have recently emerged as promising architectures for combining the efficiency of spiking neural networks with the representational power of self-attention. However, the lack of standardized implementations, evaluation pipelines, and consistent design choices has hindered fair comparison and principled analysis. In this paper, we introduce STEP a unified benchmark framework for Spiking Transformers that supports a wide range of tasks, including classification, segmentation, and detection across static, event-based, and sequential datasets. STEP provides modular support for diverse components such as spiking neurons, input encodings, surrogate gradients, and multiple backends (e.g., SpikingJelly, BrainCog). Using STEP, we reproduce and evaluate several representative models, and conduct systematic ablation studies on attention design, neuron types, encoding schemes, and temporal modeling capabilities. We also propose a unified analytical model for energy estimation, accounting for spike sparsity, bitwidth, and memory access, and show that quantized ANNs may offer comparable or better energy efficiency. Our results suggest that current Spiking Transformers rely heavily on convolutional frontends and lack strong temporal modeling, underscoring the need for spike-native architectural innovations. The full code is available at: https://github.com/Fancyssc/STEP