Impact of white Gaussian internal noise on analog echo-state neural networks

TL;DR

This paper investigates how internal white Gaussian noise affects the performance of analog echo-state neural networks, revealing that noise propagation depends on the reservoir's connection matrix properties and can severely impair signal integrity.

Contribution

It provides a detailed analysis of noise impact on ESNs with various topologies and activation functions, identifying conditions where minimal noise can destroy useful signals.

Findings

Noise propagation is mainly controlled by the mean and mean square of the connection matrix.

Certain network configurations are highly sensitive, with noise levels as low as 10^{-20} causing signal loss.

The criticality of noise varies with activation functions and network self-closure.

Abstract

In recent years, more and more works have appeared devoted to the analog (hardware) implementation of artificial neural networks, in which neurons and the connection between them are based not on computer calculations, but on physical principles. Such networks offer improved energy efficiency and, in some cases, scalability, but may be susceptible to internal noise. This paper studies the influence of noise on the functioning of recurrent networks using the example of trained echo state networks (ESNs). The most common reservoir connection matrices were chosen as various topologies of ESNs: random uniform and band matrices with different connectivity. White Gaussian noise was chosen as the influence, and according to the way of its introducing it was additive or multiplicative, as well as correlated or uncorrelated. In the paper, we show that the propagation of noise in reservoir is mainly controlled by the statistical properties of the output connection matrix, namely the mean and the mean square. Depending on these values, more correlated or uncorrelated noise accumulates in the network. We also show that there are conditions under which even noise with an intensity of $10^{-20}$ is already enough to completely lose the useful signal. In the article we show which types of noise are most critical for networks with different activation functions (hyperbolic tangent, sigmoid and linear) and if the network is self-closed.