IKUN: Initialization to Keep snn training and generalization great with sUrrogate-stable variaNce

TL;DR

IKUN is a novel initialization method for spiking neural networks that stabilizes training, accelerates convergence, and improves generalization by maintaining surrogate gradient variance, outperforming traditional initializations.

Contribution

The paper introduces IKUN, a variance-stabilizing initialization tailored for SNNs that enhances training stability and generalization, addressing limitations of traditional methods.

Findings

IKUN improves training efficiency by up to 50%.

IKUN achieves 95% training accuracy and 91% generalization accuracy.

Models trained with IKUN converge to flatter minima, aiding generalization.

Abstract

Weight initialization significantly impacts the convergence and performance of neural networks. While traditional methods like Xavier and Kaiming initialization are widely used, they often fall short for spiking neural networks (SNNs), which have distinct requirements compared to artificial neural networks (ANNs). To address this, we introduce \textbf{IKUN}, a variance-stabilizing initialization method integrated with surrogate gradient functions, specifically designed for SNNs. \textbf{IKUN} stabilizes signal propagation, accelerates convergence, and enhances generalization. Experiments show \textbf{IKUN} improves training efficiency by up to \textbf{50\%}, achieving \textbf{95\%} training accuracy and \textbf{91\%} generalization accuracy. Hessian analysis reveals that \textbf{IKUN}-trained models converge to flatter minima, characterized by Hessian eigenvalues near zero on the positive side, promoting better generalization. The method is open-sourced for further exploration: \href{https://github.com/MaeChd/SurrogateVarStabe}{https://github.com/MaeChd/SurrogateVarStabe}.