Layerwise Noise Maximisation to Train Low-Energy Deep Neural Networks

TL;DR

This paper introduces a training method that optimizes layerwise memory reliability in deep neural networks, significantly reducing energy consumption during inference with minimal impact on accuracy.

Contribution

It proposes a novel layerwise reliability optimization algorithm that reduces memory energy use in DNNs without substantial complexity overhead.

Findings

Memory energy consumption reduced by 3.3x at isoaccuracy

Layerwise reliability optimization improves energy efficiency

Negligible complexity overhead in training

Abstract

Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optimize the reliability of the storage separately for each layer of the network, while incurring a negligible complexity overhead compared to a conventional stochastic gradient descent training. For an exponential energy-reliability model, the proposed training approach can decrease the memory energy consumption of a DNN with binary parameters by 3.3$\times$ at isoaccuracy, compared to a reliable implementation.