Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks

TL;DR

This paper introduces a ternary spike neuron model for spiking neural networks that enhances information capacity and accuracy while maintaining energy-efficient, event-driven, and multiplication-free operation, outperforming existing methods.

Contribution

The paper proposes a novel ternary spike neuron with trainable amplitude, improving information transmission and accuracy in SNNs compared to binary spikes, while preserving efficiency.

Findings

Ternary spike neurons outperform binary spikes in accuracy.

Trainable spike amplitudes adapt to layer-specific distributions.

The method maintains energy efficiency and is validated on multiple datasets.

Abstract

The Spiking Neural Network (SNN), as one of the biologically inspired neural network infrastructures, has drawn increasing attention recently. It adopts binary spike activations to transmit information, thus the multiplications of activations and weights can be substituted by additions, which brings high energy efficiency. However, in the paper, we theoretically and experimentally prove that the binary spike activation map cannot carry enough information, thus causing information loss and resulting in accuracy decreasing. To handle the problem, we propose a ternary spike neuron to transmit information. The ternary spike neuron can also enjoy the event-driven and multiplication-free operation advantages of the binary spike neuron but will boost the information capacity. Furthermore, we also embed a trainable factor in the ternary spike neuron to learn the suitable spike amplitude, thus our SNN will adopt different spike amplitudes along layers, which can better suit the phenomenon that the membrane potential distributions are different along layers. To retain the efficiency of the vanilla ternary spike, the trainable ternary spike SNN will be converted to a standard one again via a re-parameterization technique in the inference. Extensive experiments with several popular network structures over static and dynamic datasets show that the ternary spike can consistently outperform state-of-the-art methods. Our code is open-sourced at https://github.com/yfguo91/Ternary-Spike.