Towards Scalable GPU-Accelerated SNN Training via Temporal Fusion

TL;DR

This paper introduces a temporal fusion method that significantly accelerates GPU training of Spiking Neural Networks, achieving up to 40x speedups and facilitating scalable SNN development on common hardware.

Contribution

The paper proposes a novel temporal fusion technique that enhances GPU-based SNN training speed, addressing a key bottleneck in SNN research and deployment.

Findings

Achieved up to 40x acceleration on NVIDIA A100 GPUs.

Validated effectiveness in both real and idealized training scenarios.

Demonstrated scalability on single and multi-GPU systems.

Abstract

Drawing on the intricate structures of the brain, Spiking Neural Networks (SNNs) emerge as a transformative development in artificial intelligence, closely emulating the complex dynamics of biological neural networks. While SNNs show promising efficiency on specialized sparse-computational hardware, their practical training often relies on conventional GPUs. This reliance frequently leads to extended computation times when contrasted with traditional Artificial Neural Networks (ANNs), presenting significant hurdles for advancing SNN research. To navigate this challenge, we present a novel temporal fusion method, specifically designed to expedite the propagation dynamics of SNNs on GPU platforms, which serves as an enhancement to the current significant approaches for handling deep learning tasks with SNNs. This method underwent thorough validation through extensive experiments in both authentic training scenarios and idealized conditions, confirming its efficacy and adaptability for single and multi-GPU systems. Benchmarked against various existing SNN libraries/implementations, our method achieved accelerations ranging from $5\times$ to $40\times$ on NVIDIA A100 GPUs. Publicly available experimental codes can be found at https://github.com/EMI-Group/snn-temporal-fusion.