A More Accurate Approximation of Activation Function with Few Spikes Neurons

TL;DR

This paper introduces a tendency-based parameter initialization method to improve the approximation accuracy of activation functions using few-spike neurons in spiking neural networks, addressing previous limitations.

Contribution

It proposes a novel TBPI method that leverages temporal dependencies for better training of FS neurons to approximate complex activation functions.

Findings

Enhanced approximation accuracy of activation functions.

Improved training efficiency for FS neurons.

Potential energy savings in neural network implementations.

Abstract

Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters.