BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation

TL;DR

This paper introduces a blurred knowledge distillation technique to improve the accuracy of learning-based spiking neural networks, achieving state-of-the-art results on static and neuromorphic datasets.

Contribution

The paper proposes a novel blurred knowledge distillation method that enhances SNN training by leveraging blurred features and combining with logits-based knowledge distillation.

Findings

Achieves state-of-the-art accuracy on ImageNet with CNN and Transformer SNNs.

Outperforms previous methods by 4.51% and 0.93% on ImageNet.

Effective for both static and neuromorphic datasets.

Abstract

Spiking neural networks (SNNs), which mimic biological neural system to convey information via discrete spikes, are well known as brain-inspired models with excellent computing efficiency. By utilizing the surrogate gradient estimation for discrete spikes, learning-based SNN training methods that can achieve ultra-low inference latency (number of time-step) emerge recently. Nevertheless, due to the difficulty in deriving precise gradient estimation for discrete spikes using learning-based method, a distinct accuracy gap persists between SNN and its artificial neural networks (ANNs) counterpart. To address the aforementioned issue, we propose a blurred knowledge distillation (BKD) technique, which leverages random blurred SNN feature to restore and imitate the ANN feature. Note that, our BKD is applied upon the feature map right before the last layer of SNN, which can also mix with prior logits-based knowledge distillation for maximized accuracy boost. To our best knowledge, in the category of learning-based methods, our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets. On ImageNet dataset, BKDSNN outperforms prior best results by 4.51% and 0.93% with the network topology of CNN and Transformer respectively.