Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication

TL;DR

Whetstone is a novel training method that enables deep neural networks to operate with binary spike-like communication, suitable for neuromorphic hardware, without significant performance loss.

Contribution

The paper introduces Whetstone, a new iterative backpropagation-based method for training deep spiking neural networks with binary communication.

Findings

Whetstone effectively configures networks for spiking hardware with minimal performance loss.

Networks trained with Whetstone use single time step binary communication, comparable to conventional ANNs.

Whetstone is compatible with various neural network applications beyond classification.

Abstract

This paper presents a new technique for training networks for low-precision communication. Targeting minimal communication between nodes not only enables the use of emerging spiking neuromorphic platforms, but may additionally streamline processing conventionally. Low-power and embedded neuromorphic processors potentially offer dramatic performance-per-Watt improvements over traditional von Neumann processors, however programming these brain-inspired platforms generally requires platform-specific expertise which limits their applicability. To date, the majority of artificial neural networks have not operated using discrete spike-like communication. We present a method for training deep spiking neural networks using an iterative modification of the backpropagation optimization algorithm. This method, which we call Whetstone, effectively and reliably configures a network for a spiking hardware target with little, if any, loss in performance. Whetstone networks use single time step binary communication and do not require a rate code or other spike-based coding scheme, thus producing networks comparable in timing and size to conventional ANNs, albeit with binarized communication. We demonstrate Whetstone on a number of image classification networks, describing how the sharpening process interacts with different training optimizers and changes the distribution of activity within the network. We further note that Whetstone is compatible with several non-classification neural network applications, such as autoencoders and semantic segmentation. Whetstone is widely extendable and currently implemented using custom activation functions within the Keras wrapper to the popular TensorFlow machine learning framework.