Understanding and mitigating noise in trained deep neural networks

TL;DR

This paper analyzes how noise propagates in deep neural networks with analog neurons, providing methods to predict and suppress noise, which is crucial for designing energy-efficient, noise-resilient hardware.

Contribution

It introduces the first analytical framework for noise propagation in trained deep neural networks with nonlinear neurons, highlighting conditions to suppress noise accumulation.

Findings

Noise propagation is generally bounded and does not worsen with more layers.

Neuron activation functions with slopes less than one can completely suppress noise accumulation.

The framework aids in designing noise-resilient analog neural network hardware.

Abstract

Deep neural networks unlocked a vast range of new applications by solving tasks of which many were previously deemed as reserved to higher human intelligence. One of the developments enabling this success was a boost in computing power provided by special purpose hardware, such as graphic or tensor processing units. However, these do not leverage fundamental features of neural networks like parallelism and analog state variables. Instead, they emulate neural networks relying on binary computing, which results in unsustainable energy consumption and comparatively low speed. Fully parallel and analogue hardware promises to overcome these challenges, yet the impact of analogue neuron noise and its propagation, i.e. accumulation, threatens rendering such approaches inept. Here, we determine for the first time the propagation of noise in deep neural networks comprising noisy nonlinear neurons in trained fully connected layers. We study additive and multiplicative as well as correlated and uncorrelated noise, and develop analytical methods that predict the noise level in any layer of symmetric deep neural networks or deep neural networks trained with back propagation. We find that noise accumulation is generally bound, and adding additional network layers does not worsen the signal to noise ratio beyond a limit. Most importantly, noise accumulation can be suppressed entirely when neuron activation functions have a slope smaller than unity. We therefore developed the framework for noise in fully connected deep neural networks implemented in analog systems, and identify criteria allowing engineers to design noise-resilient novel neural network hardware.