High-Accuracy Inference in Neuromorphic Circuits using Hardware-Aware Training

TL;DR

This paper introduces a hardware-aware training algorithm for neuromorphic circuits that maintains high inference accuracy despite hardware non-idealities, enabling efficient edge device neural network deployment.

Contribution

It proposes a novel off-line, hardware-aware training method applicable to various hardware models, improving accuracy of low-bitwidth neural networks on neuromorphic hardware.

Findings

Negligible accuracy loss with hardware-aware training on MNIST and EMNIST datasets.

Applicable to a wide range of hardware models and neural network architectures.

Uses standard neural network training methods with hardware-specific modifications.

Abstract

Neuromorphic Multiply-And-Accumulate (MAC) circuits utilizing synaptic weight elements based on SRAM or novel Non-Volatile Memories (NVMs) provide a promising approach for highly efficient hardware representations of neural networks. NVM density and robustness requirements suggest that off-line training is the right choice for "edge" devices, since the requirements for synapse precision are much less stringent. However, off-line training using ideal mathematical weights and activations can result in significant loss of inference accuracy when applied to non-ideal hardware. Non-idealities such as multi-bit quantization of weights and activations, non-linearity of weights, finite max/min ratios of NVM elements, and asymmetry of positive and negative weight components all result in degraded inference accuracy. In this work, it is demonstrated that non-ideal Multi-Layer Perceptron (MLP) architectures using low bitwidth weights and activations can be trained with negligible loss of inference accuracy relative to their Floating Point-trained counterparts using a proposed off-line, continuously differentiable HW-aware training algorithm. The proposed algorithm is applicable to a wide range of hardware models, and uses only standard neural network training methods. The algorithm is demonstrated on the MNIST and EMNIST datasets, using standard MLPs.