SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling

TL;DR

This paper introduces inter-layer pipelining and multiprocessor scheduling techniques to significantly accelerate the training of Spiking Neural Networks, achieving up to 2X speedup with minimal communication overhead.

Contribution

It is the first to propose inter-layer pipelining for SNN training using systolic arrays and multiprocessor scheduling, improving training speed without degrading accuracy.

Findings

Achieves an average of 1.6X speedup over standard pipelining.

Up to 2X speedup in some cases.

Communication overhead is less than 0.5% of total training communication.

Abstract

Spiking Neural Networks (SNNs) have gained popularity due to their high energy efficiency. Prior works have proposed various methods for training SNNs, including backpropagation-based methods. Training SNNs is computationally expensive compared to their conventional counterparts and would benefit from multiprocessor hardware acceleration. This is the first paper to propose inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling. The impact of training using delayed gradients is observed using three networks training on different datasets, showing no degradation for small networks and < 10% degradation for large networks. The mapping of various training tasks of the SNN onto systolic arrays is formulated, and the proposed scheduling method is evaluated on the three networks. The results are compared against standard pipelining algorithms. The results show that the proposed method achieves an average speedup of 1.6X compared to standard pipelining algorithms, with an upwards of 2X improvement in some cases. The incurred communication overhead due to the proposed method is less than 0.5% of the total required communication of training.