RTFormer: Re-parameter TSBN Spiking Transformer
Hongzhi Wang, Xiubo Liang, Mengjian Li, Tao Zhang
TL;DR
RTFormer is a novel spiking neural network architecture that combines re-parameterized convolutions and TSBN to optimize energy efficiency without sacrificing computational performance.
Contribution
It introduces RTFormer, integrating re-parameterized TSBN within Spiking Transformers to enhance energy efficiency and computational robustness.
Findings
Improved energy efficiency during inference.
Maintains high computational performance.
Balances energy use with task complexity.
Abstract
The Spiking Neural Networks (SNNs), renowned for their bio-inspired operational mechanism and energy efficiency, mirror the human brain's neural activity. Yet, SNNs face challenges in balancing energy efficiency with the computational demands of advanced tasks. Our research introduces the RTFormer, a novel architecture that embeds Re-parameterized Temporal Sliding Batch Normalization (TSBN) within the Spiking Transformer framework. This innovation optimizes energy usage during inference while ensuring robust computational performance. The crux of RTFormer lies in its integration of reparameterized convolutions and TSBN, achieving an equilibrium between computational prowess and energy conservation.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Fiber Laser Technologies · Semiconductor Lasers and Optical Devices
MethodsLinear Layer · Multi-Head Attention · Residual Connection · Softmax · Layer Normalization · Byte Pair Encoding · Label Smoothing · Batch Normalization · Position-Wise Feed-Forward Layer · Dropout