On Hardening DNNs against Noisy Computations

TL;DR

This paper explores methods to improve the robustness of deep neural networks against noisy analog computations, finding that noisy training generally outperforms quantization-aware training in maintaining accuracy under noise.

Contribution

It compares quantization-aware training and noisy training for enhancing neural network robustness against analog computation noise, highlighting the superiority of noisy training.

Findings

Noisy training outperforms quantization-aware training in robustness.

Quantization-aware training with constant scaling improves noise tolerance.

Robustness gains are consistent across various neural network architectures.

Abstract

The success of deep learning has sparked significant interest in designing computer hardware optimized for the high computational demands of neural network inference. As further miniaturization of digital CMOS processors becomes increasingly challenging, alternative computing paradigms, such as analog computing, are gaining consideration. Particularly for compute-intensive tasks such as matrix multiplication, analog computing presents a promising alternative due to its potential for significantly higher energy efficiency compared to conventional digital technology. However, analog computations are inherently noisy, which makes it challenging to maintain high accuracy on deep neural networks. This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise. Experimental results across various network architectures show that quantization-aware training with constant scaling factors enhances robustness. We compare these methods with noisy training, which incorporates a noise injection during training that mimics the noise encountered during inference. While both two methods increase tolerance against noise, noisy training emerges as the superior approach for achieving robust neural network performance, especially in complex neural architectures.