Noisy Computations during Inference: Harmful or Helpful?

TL;DR

This paper explores how noise during neural network inference can be mitigated to improve accuracy and also utilized to defend against adversarial attacks, with promising results in power-efficient neuromorphic systems.

Contribution

It introduces noise-injected training and voting methods to significantly improve inference accuracy under noisy conditions and demonstrates noise's potential as a defense against black-box adversarial attacks.

Findings

Inference accuracy improved from 21.1% to 99.5% on MNIST with noise.

Robustness against black-box attacks increased by over 0.5%.

Noise-injected training enables neural networks to be noise-insensitive.

Abstract

We study two aspects of noisy computations during inference. The first aspect is how to mitigate their side effects for naturally trained deep learning systems. One of the motivations for looking into this problem is to reduce the high power cost of conventional computing of neural networks through the use of analog neuromorphic circuits. Traditional GPU/CPU-centered deep learning architectures exhibit bottlenecks in power-restricted applications (e.g., embedded systems). The use of specialized neuromorphic circuits, where analog signals passed through memory-cell arrays are sensed to accomplish matrix-vector multiplications, promises large power savings and speed gains but brings with it the problems of limited precision of computations and unavoidable analog noise. We manage to improve inference accuracy from 21.1% to 99.5% for MNIST images, from 29.9% to 89.1% for CIFAR10, and from 15.5% to 89.6% for MNIST stroke sequences with the presence of strong noise (with signal-to-noise power ratio being 0 dB) by noise-injected training and a voting method. This observation promises neural networks that are insensitive to inference noise, which reduces the quality requirements on neuromorphic circuits and is crucial for their practical usage. The second aspect is how to utilize the noisy inference as a defensive architecture against black-box adversarial attacks. During inference, by injecting proper noise to signals in the neural networks, the robustness of adversarially-trained neural networks against black-box attacks has been further enhanced by 0.5% and 1.13% for two adversarially trained models for MNIST and CIFAR10, respectively.