An Exact Mapping From ReLU Networks to Spiking Neural Networks

TL;DR

This paper presents an exact method to convert deep ReLU neural networks into spiking neural networks that fire one spike per neuron, achieving zero accuracy loss on multiple datasets, thus enabling energy-efficient AI.

Contribution

The authors introduce a constructive proof for an exact mapping from deep ReLU networks to spiking neural networks with no performance degradation.

Findings

Zero percent accuracy drop on CIFAR10, CIFAR100, Places365, and PASS datasets.

The mapping applies to networks with convolutional layers, batch normalization, and max pooling.

Deep ReLU networks can be replaced by single-spike SNNs for energy-efficient AI.

Abstract

Deep spiking neural networks (SNNs) offer the promise of low-power artificial intelligence. However, training deep SNNs from scratch or converting deep artificial neural networks to SNNs without loss of performance has been a challenge. Here we propose an exact mapping from a network with Rectified Linear Units (ReLUs) to an SNN that fires exactly one spike per neuron. For our constructive proof, we assume that an arbitrary multi-layer ReLU network with or without convolutional layers, batch normalization and max pooling layers was trained to high performance on some training set. Furthermore, we assume that we have access to a representative example of input data used during training and to the exact parameters (weights and biases) of the trained ReLU network. The mapping from deep ReLU networks to SNNs causes zero percent drop in accuracy on CIFAR10, CIFAR100 and the ImageNet-like data sets Places365 and PASS. More generally our work shows that an arbitrary deep ReLU network can be replaced by an energy-efficient single-spike neural network without any loss of performance.