Inherent Redundancy in Spiking Neural Networks

TL;DR

This paper investigates the inherent redundancy in Spiking Neural Networks caused by spatio-temporal invariance, analyzes its effects, and proposes an attention-based method to optimize spike firing and improve performance.

Contribution

It introduces an Advance Spatial Attention (ASA) module that adaptively reduces noise spikes and enhances SNN efficiency, a novel approach to redundancy management in SNNs.

Findings

Significant reduction in spike firing with improved accuracy.

The ASA module outperforms state-of-the-art SNN baselines.

Enhanced parameter utilization through spatial attention.

Abstract

Spiking Neural Networks (SNNs) are well known as a promising energy-efficient alternative to conventional artificial neural networks. Subject to the preconceived impression that SNNs are sparse firing, the analysis and optimization of inherent redundancy in SNNs have been largely overlooked, thus the potential advantages of spike-based neuromorphic computing in accuracy and energy efficiency are interfered. In this work, we pose and focus on three key questions regarding the inherent redundancy in SNNs. We argue that the redundancy is induced by the spatio-temporal invariance of SNNs, which enhances the efficiency of parameter utilization but also invites lots of noise spikes. Further, we analyze the effect of spatio-temporal invariance on the spatio-temporal dynamics and spike firing of SNNs. Then, motivated by these analyses, we propose an Advance Spatial Attention (ASA) module to harness SNNs' redundancy, which can adaptively optimize their membrane potential distribution by a pair of individual spatial attention sub-modules. In this way, noise spike features are accurately regulated. Experimental results demonstrate that the proposed method can significantly drop the spike firing with better performance than state-of-the-art SNN baselines. Our code is available in \url{https://github.com/BICLab/ASA-SNN}.