Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration

TL;DR

This paper introduces NDSNN, a training acceleration framework for Spiking Neural Networks inspired by neurogenesis dynamics, achieving high sparsity, improved accuracy, and reduced training costs on standard datasets.

Contribution

The paper proposes a novel neurogenesis-inspired dynamic sparsity training method for SNNs that enhances training efficiency without sacrificing model fidelity.

Findings

Achieves up to 20.52% accuracy improvement on TinyImageNet with 99% sparsity.

Reduces training cost to 40.89% of LTH on ResNet-19.

Maintains high accuracy with extreme sparsity through a new drop-and-grow strategy.

Abstract

Biologically inspired Spiking Neural Networks (SNNs) have attracted significant attention for their ability to provide extremely energy-efficient machine intelligence through event-driven operation and sparse activities. As artificial intelligence (AI) becomes ever more democratized, there is an increasing need to execute SNN models on edge devices. Existing works adopt weight pruning to reduce SNN model size and accelerate inference. However, these methods mainly focus on how to obtain a sparse model for efficient inference, rather than training efficiency. To overcome these drawbacks, in this paper, we propose a Neurogenesis Dynamics-inspired Spiking Neural Network training acceleration framework, NDSNN. Our framework is computational efficient and trains a model from scratch with dynamic sparsity without sacrificing model fidelity. Specifically, we design a new drop-and-grow strategy with decreasing number of non-zero weights, to maintain extreme high sparsity and high accuracy. We evaluate NDSNN using VGG-16 and ResNet-19 on CIFAR-10, CIFAR-100 and TinyImageNet. Experimental results show that NDSNN achieves up to 20.52\% improvement in accuracy on Tiny-ImageNet using ResNet-19 (with a sparsity of 99\%) as compared to other SOTA methods (e.g., Lottery Ticket Hypothesis (LTH), SET-SNN, RigL-SNN). In addition, the training cost of NDSNN is only 40.89\% of the LTH training cost on ResNet-19 and 31.35\% of the LTH training cost on VGG-16 on CIFAR-10.