Exploring Temporal Information Dynamics in Spiking Neural Networks

TL;DR

This paper investigates how temporal information concentrates in Spiking Neural Networks during training, revealing its impact on robustness and proposing a pruning method based on this phenomenon.

Contribution

It provides the first empirical analysis of temporal information dynamics in SNNs, introduces a loss function to manipulate this concentration, and proposes a pruning method leveraging this insight.

Findings

Temporal information concentrates in early timesteps during training.

Concentration phenomenon is consistent across various configurations.

Manipulating temporal information affects robustness but not accuracy.

Abstract

Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.