Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

TL;DR

This paper introduces a novel logit-based knowledge distillation method for deep Spiking Neural Networks that maintains high accuracy across all inference timesteps without retraining, improving deployment flexibility and efficiency.

Contribution

It proposes a new distillation framework leveraging the spatio-temporal properties of SNNs to optimize performance over full-range timesteps without retraining.

Findings

Achieves state-of-the-art results on multiple datasets

Ensures convergence of models across all timesteps

Enhances deployment flexibility of SNNs

Abstract

Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods. Our code is available at https://github.com/Intelli-Chip-Lab/snn\_temporal\_decoupling\_distillation.