Training Spiking Neural Networks with Local Tandem Learning

TL;DR

This paper introduces Local Tandem Learning, a novel training method for deep spiking neural networks that mimics pre-trained ANNs, enabling rapid training, high accuracy, and hardware-friendly implementation for neuromorphic chips.

Contribution

The paper proposes a generalized, layer-wise learning rule called Local Tandem Learning that accelerates training and improves hardware compatibility of deep SNNs.

Findings

Achieves comparable accuracy to ANNs on CIFAR-10, CIFAR-100, and Tiny ImageNet.

Rapid convergence within five epochs on CIFAR-10.

Hardware-friendly implementation suitable for neuromorphic chips.

Abstract

Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient over their predecessors. However, there is a lack of an efficient and generalized training method for deep SNNs, especially for deployment on analog computing substrates. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL). The LTL rule follows the teacher-student learning approach by mimicking the intermediate feature representations of a pre-trained ANN. By decoupling the learning of network layers and leveraging highly informative supervisor signals, we demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity. Our experimental results have also shown that the SNNs thus trained can achieve comparable accuracies to their teacher ANNs on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Moreover, the proposed LTL rule is hardware friendly. It can be easily implemented on-chip to perform fast parameter calibration and provide robustness against the notorious device non-ideality issues. It, therefore, opens up a myriad of opportunities for training and deployment of SNN on ultra-low-power mixed-signal neuromorphic computing chips.10