TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

TL;DR

TEFormer introduces a novel bidirectional temporal fusion mechanism in Spiking Transformers, significantly enhancing their ability to model spatiotemporal dependencies and outperform existing models across diverse benchmarks.

Contribution

It is the first framework to incorporate bidirectional temporal fusion in Spiking Transformers, combining parallel attention-based fusion with backward recurrent aggregation.

Findings

Consistently outperforms baseline SNN and Spiking Transformer models.

Performance remains stable across different neural encoding schemes.

Establishes TEFormer as a general framework for temporal modeling in SNNs.

Abstract

In recent years, Spiking Neural Networks (SNNs) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling. However, existing Spiking Transformers still lack a principled mechanism for effective temporal fusion, limiting their ability to fully exploit spatiotemporal dependencies. Inspired by feedforward-feedback modulation in the human visual pathway, we propose TEFormer, the first Spiking Transformer framework that achieves bidirectional temporal fusion by decoupling temporal modeling across its core components. Specifically, TEFormer employs a lightweight and hyperparameter-free forward temporal fusion mechanism in the attention module, enabling fully parallel computation, while incorporating a backward gated recurrent structure in the MLP to aggregate temporal information in reverse order and reinforce temporal consistency. Extensive experiments across a wide range of benchmarks demonstrate that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines under diverse datasets. Moreover, through the first systematic evaluation of Spiking Transformers under different neural encoding schemes, we show that the performance gains of TEFormer remain stable across encoding choices, indicating that the improved temporal modeling directly translates into reliable accuracy improvements across varied spiking representations. These results collectively establish TEFormer as an effective and general framework for temporal modeling in Spiking Transformers.