Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation

TL;DR

This paper investigates the impact of noise in analog neural network hardware and proposes a distillation-based training method to significantly improve noise robustness, facilitating practical deployment of analog accelerators.

Contribution

It introduces a novel approach combining knowledge distillation and noise injection to enhance neural network robustness against hardware noise.

Findings

Achieves up to two times greater noise tolerance than previous methods.

Demonstrates effectiveness across multiple networks and datasets, including ImageNet.

Provides insights into the reduced capacity of noisy neural networks due to information loss.

Abstract

The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference. In particular, analog computing hardware has been heavily motivated specifically for accelerating neural networks, based on either electronic, optical or photonic devices, which may well achieve lower power consumption than conventional digital electronics. However, these proposed analog accelerators suffer from the intrinsic noise generated by their physical components, which makes it challenging to achieve high accuracy on deep neural networks. Hence, for successful deployment on analog accelerators, it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights, which is a somewhat new challenge in machine learning. In this paper, we advance the understanding of noisy neural networks. We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output. To combat this, we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks, which is demonstrated experimentally across different networks and datasets, including ImageNet. Our method achieves models with as much as two times greater noise tolerance compared with the previous best attempts, which is a significant step towards making analog hardware practical for deep learning.