Temporal Regularization Training: Unleashing the Potential of Spiking Neural Networks

TL;DR

This paper introduces a temporal regularization training method for Spiking Neural Networks that enhances their performance and generalization by focusing on early timesteps and mitigating issues like gradient vanishing and overfitting.

Contribution

The paper proposes a novel TRT method with a time-decaying regularization mechanism, providing theoretical analysis and empirical validation for improved SNN training.

Findings

TRT mitigates temporal gradient vanishing in SNNs.

TRT helps SNNs converge to flatter minima with better generalization.

Experimental results show TRT improves performance on static and neuromorphic datasets.

Abstract

Spiking Neural Networks (SNNs) have received widespread attention due to their event-driven and low-power characteristics, making them particularly effective for processing neuromorphic data. Recent studies have shown that directly trained SNNs suffer from severe temporal gradient vanishing and overfitting issues, which fundamentally constrain their performance and generalizability. This paper unveils a temporal regularization training (TRT) memthod, designed to unleash the generalization and performance potential of SNNs through a time-decaying regularization mechanism that prioritizes early timesteps with stronger constraints. We perform theoretical analysis to reveal TRT's ability on mitigating the temporal gradient vanishment. To validate the effectiveness of TRT, we conduct experiments on both static image datasets and dynamic neuromorphic datasets, perform analysis of their results, demonstrating that TRT can effectively mitigate overfitting and help SNNs converge into flatter local minima with better generalizability. Furthermore, we establish a theoretical interpretation of TRT's temporal regularization mechanism by analyzing the temporal information dynamics inside SNNs. We track the Fisher information of SNNs during training process, showing that Fisher information progressively concentrates in early timesteps. The time-decaying regularization mechanism implemented in TRT effectively guides the network to learn robust features in early timesteps with rich information, thereby leading to significant improvements in model generalization.