Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation

TL;DR

This paper introduces a novel downsampling module, CML, for deep spiking neural networks that enables precise gradient backpropagation, leading to state-of-the-art performance across multiple datasets.

Contribution

The paper proposes CML, an SNN-optimized downsampling method that addresses gradient imprecision, with theoretical proof and extensive empirical validation showing improved results.

Findings

Achieves state-of-the-art accuracy on ImageNet and CIFAR datasets.

Significantly improves performance over Spikingformer.

Provides theoretical proof of effective gradient backpropagation.

Abstract

Deep spiking neural networks (SNNs) have drawn much attention in recent years because of their low power consumption, biological rationality and event-driven property. However, state-of-the-art deep SNNs (including Spikformer and Spikingformer) suffer from a critical challenge related to the imprecise gradient backpropagation. This problem arises from the improper design of downsampling modules in these networks, and greatly hampering the overall model performance. In this paper, we propose ConvBN-MaxPooling-LIF (CML), an SNN-optimized downsampling with precise gradient backpropagation. We prove that CML can effectively overcome the imprecision of gradient backpropagation from a theoretical perspective. In addition, we evaluate CML on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS, DVS128-Gesture datasets, and show state-of-the-art performance on all these datasets with significantly enhanced performances compared with Spikingformer. For instance, our model achieves 77.64 $\%$ on ImageNet, 96.04 $\%$ on CIFAR10, 81.4$\%$ on CIFAR10-DVS, with + 1.79$\%$ on ImageNet, +1.16$\%$ on CIFAR100 compared with Spikingformer.