# Adaptive and lightweight surrogate gradients: enhancing training efficiency of spiking neural networks

## TL;DR

This paper introduces a new method called AdaLi to improve the training of energy-efficient spiking neural networks by addressing common issues like gradient mismatch and instability.

## Contribution

AdaLi introduces adaptive and lightweight surrogate gradients to enhance training efficiency and stability in spiking neural networks.

## Key findings

- AdaLi outperforms baseline methods in training efficiency and accuracy on static and neuromorphic datasets.
- The method effectively mitigates gradient vanishing and explosion through stable surrogate gradients.
- AdaLi provides hyperparameters that can be manually or automatically adjusted to address gradient mismatch.

## Abstract

Spiking Neural Networks (SNNs) have emerged as a promising paradigm in artificial intelligence due to their energy efficiency. However, training SNNs remains a formidable challenge because the nondifferentiable nature of spike activation functions prevents the direct application of conventional backpropagation. Existing surrogate gradient methods often suffer from critical limitations, including gradient mismatch, gradient explosion or vanishing, and high computational overhead.

In this paper, we propose an Adaptive and Lightweight (AdaLi) backpropagation method to address these issues. AdaLi reduces the computational complexity of the training process by introducing lightweight surrogate gradients and dynamically adjusting gradient update boundaries. Furthermore, it employs an adaptive mechanism to adjust the surrogate gradients based on training epochs, thereby enhancing network stability. The method also provides additional hyperparameters to address gradient mismatch, which can be either manually fine-tuned or automatically determined based on the distribution of spiking neuron membrane potentials.

Experimental results on both static and neuromorphic datasets demonstrate that SNNs trained with AdaLi outperform their baseline counterparts in terms of efficiency and accuracy. The stable surrogate gradients in AdaLi effectively mitigate the issues of gradient vanishing and explosion.

The AdaLi method introduces a novel approach to optimizing gradient calculation and parameter updates in SNNs, paving the way for more effective and accurate training. The source code is available at https://github.com/parania/AdaLi.