The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks

TL;DR

This paper investigates how quantization and pruning techniques affect the performance and energy efficiency of spiking neural networks on hardware, revealing that aggressive quantization is highly effective while pruning has limitations.

Contribution

It provides a comprehensive analysis of the combined effects of quantization and pruning on SNNs, highlighting their impact on accuracy and energy consumption on specialized hardware.

Findings

Quantization down to ternary weights does not reduce accuracy.

Pruning maintains accuracy up to 80% sparsity but increases energy consumption.

Combining pruning and quantization offers a trade-off between energy efficiency and accuracy.

Abstract

Energy efficient implementations and deployments of Spiking neural networks (SNNs) have been of great interest due to the possibility of developing artificial systems that can achieve the computational powers and energy efficiency of the biological brain. Efficient implementations of SNNs on modern digital hardware are also inspired by advances in machine learning and deep neural networks (DNNs). Two techniques widely employed in the efficient deployment of DNNs -- the quantization and pruning of parameters, can both compress the model size, reduce memory footprints, and facilitate low-latency execution. The interaction between quantization and pruning and how they might impact model performance on SNN accelerators is currently unknown. We study various combinations of pruning and quantization in isolation, cumulatively, and simultaneously (jointly) to a state-of-the-art SNN targeting gesture recognition for dynamic vision sensor cameras (DVS). We show that this state-of-the-art model is amenable to aggressive parameter quantization, not suffering from any loss in accuracy down to ternary weights. However, pruning only maintains iso-accuracy up to 80% sparsity, which results in 45% more energy than the best quantization on our architectural model. Applying both pruning and quantization can result in an accuracy loss to offer a favourable trade-off on the energy-accuracy Pareto-frontier for the given hardware configuration.