Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks

TL;DR

This paper introduces a cost-effective self-distillation method for spiking neural networks that leverages temporal and spatial consistency during training, improving performance without additional inference costs.

Contribution

The paper proposes a novel temporal-spatial self-distillation approach for SNNs that eliminates the need for external teacher models and enhances learning efficiency.

Findings

Achieves superior accuracy on CIFAR10/100 and ImageNet datasets.

Improves neuromorphic dataset performance on CIFAR10-DVS and DVS-Gesture.

No additional inference overhead introduced.

Abstract

Spiking neural networks (SNNs) have attracted considerable attention for their event-driven, low-power characteristics and high biological interpretability. Inspired by knowledge distillation (KD), recent research has improved the performance of the SNN model with a pre-trained teacher model. However, additional teacher models require significant computational resources, and it is tedious to manually define the appropriate teacher network architecture. In this paper, we explore cost-effective self-distillation learning of SNNs to circumvent these concerns. Without an explicit defined teacher, the SNN generates pseudo-labels and learns consistency during training. On the one hand, we extend the timestep of the SNN during training to create an implicit temporal ``teacher" that guides the learning of the original ``student", i.e., the temporal self-distillation. On the other hand, we guide the output of the weak classifier at the intermediate stage by the final output of the SNN, i.e., the spatial self-distillation. Our temporal-spatial self-distillation (TSSD) learning method does not introduce any inference overhead and has excellent generalization ability. Extensive experiments on the static image datasets CIFAR10/100 and ImageNet as well as the neuromorphic datasets CIFAR10-DVS and DVS-Gesture validate the superior performance of the TSSD method. This paper presents a novel manner of fusing SNNs with KD, providing insights into high-performance SNN learning methods.