On Reducing Activity with Distillation and Regularization for Energy Efficient Spiking Neural Networks

TL;DR

This paper introduces methods to train spiking neural networks that maintain high accuracy while significantly reducing spiking activity, thereby enhancing their energy efficiency for edge applications.

Contribution

It proposes a novel approach combining Knowledge Distillation and Logits Regularization to reduce spiking activity without sacrificing performance in SNNs.

Findings

Spiking activity reduced by up to 26.73% on GSC dataset.

Spiking activity reduced by up to 14.32% on CIFAR-10.

Accuracy preserved despite activity reduction.

Abstract

Interest in spiking neural networks (SNNs) has been growing steadily, promising an energy-efficient alternative to formal neural networks (FNNs), commonly known as artificial neural networks (ANNs). Despite increasing interest, especially for Edge applications, these event-driven neural networks suffered from their difficulty to be trained compared to FNNs. To alleviate this problem, a number of innovative methods have been developed to provide performance more or less equivalent to that of FNNs. However, the spiking activity of a network during inference is usually not considered. While SNNs may usually have performance comparable to that of FNNs, it is often at the cost of an increase of the network's activity, thus limiting the benefit of using them as a more energy-efficient solution. In this paper, we propose to leverage Knowledge Distillation (KD) for SNNs training with surrogate gradient descent in order to optimize the trade-off between performance and spiking activity. Then, after understanding why KD led to an increase in sparsity, we also explored Activations regularization and proposed a novel method with Logits Regularization. These approaches, validated on several datasets, clearly show a reduction in network spiking activity (-26.73% on GSC and -14.32% on CIFAR-10) while preserving accuracy.