Efficient Training of Spiking Neural Networks with Temporally-Truncated Local Backpropagation through Time

TL;DR

This paper introduces an efficient training algorithm for spiking neural networks that combines local supervision with temporally-truncated BPTT, significantly reducing computational costs while maintaining or improving accuracy on certain datasets.

Contribution

It proposes a novel training method integrating local supervision with truncated BPTT, optimizing resource use and accuracy for deep SNNs.

Findings

Reduces GPU memory usage by 89.94%

Improves accuracy on CIFAR10-DVS by 7.26%

Balances accuracy loss and overfitting through local training

Abstract

Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algorithm explores both temporal and spatial locality in BPTT and contributes to significant reduction in computational cost including GPU memory utilization, main memory access and arithmetic operations. We thoroughly explore the design space concerning temporal truncation length and local training block size and benchmark their impact on classification accuracy of different networks running different types of tasks. The results reveal that temporal truncation has a negative effect on the accuracy of classifying frame-based datasets, but leads to improvement in accuracy on dynamic-vision-sensor (DVS) recorded datasets. In spite of resulting information loss, local training is capable of alleviating overfitting. The combined effect of temporal truncation and local training can lead to the slowdown of accuracy drop and even improvement in accuracy. In addition, training deep SNNs models such as AlexNet classifying CIFAR10-DVS dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to the standard end-to-end BPTT.