Self-Attentive Spatio-Temporal Calibration for Precise Intermediate Layer Matching in ANN-to-SNN Distillation

TL;DR

This paper introduces SASTC, a self-attentive method for aligning intermediate layers in ANN-to-SNN distillation, significantly improving SNN accuracy by addressing spatial and temporal semantic mismatches.

Contribution

The paper proposes a novel self-attentive spatio-temporal calibration method that effectively matches ANN and SNN layers, enhancing knowledge transfer and performance.

Findings

Achieved 95.12% on CIFAR-10 and 79.40% on CIFAR-100 with 2 time steps

Outperformed existing methods on multiple datasets including ImageNet and neuromorphic datasets

First to surpass ANNs with SNNs on CIFAR-10 and CIFAR-100

Abstract

Spiking Neural Networks (SNNs) are promising for low-power computation due to their event-driven mechanism but often suffer from lower accuracy compared to Artificial Neural Networks (ANNs). ANN-to-SNN knowledge distillation can improve SNN performance, but previous methods either focus solely on label information, missing valuable intermediate layer features, or use a layer-wise approach that neglects spatial and temporal semantic inconsistencies, leading to performance degradation.To address these limitations, we propose a novel method called self-attentive spatio-temporal calibration (SASTC). SASTC uses self-attention to identify semantically aligned layer pairs between ANN and SNN, both spatially and temporally. This enables the autonomous transfer of relevant semantic information. Extensive experiments show that SASTC outperforms existing methods, effectively solving the mismatching problem. Superior accuracy results include 95.12% on CIFAR-10, 79.40% on CIFAR-100 with 2 time steps, and 68.69% on ImageNet with 4 time steps for static datasets, and 97.92% on DVS-Gesture and 83.60% on DVS-CIFAR10 for neuromorphic datasets. This marks the first time SNNs have outperformed ANNs on both CIFAR-10 and CIFAR-100, shedding the new light on the potential applications of SNNs.