Phase transitions from linear to nonlinear information processing in neural networks

TL;DR

This paper explores a phase transition in echo state networks from linear to nonlinear processing, revealing a sharp capacity increase at a critical point and establishing a scaling law relating nonlinearity to noise.

Contribution

It identifies a novel discontinuous phase transition in reservoir computing networks and characterizes its dependence on noise and network size.

Findings

Capacity increases sharply beyond a critical nonlinearity threshold.

The transition becomes sharper as network size grows.

Critical nonlinearity scales with noise intensity, vanishing without noise.

Abstract

We investigate a phase transition from linear to nonlinear information processing in echo state networks, a widely used framework in reservoir computing. The network consists of randomly connected recurrent nodes perturbed by a noise and the output is obtained through linear regression on the network states. By varying the standard deviation of the input weights, we systematically control the nonlinearity of the network. For small input standard deviations, the network operates in an approximately linear regime, resulting in limited information processing capacity. However, beyond a critical threshold, the capacity increases rapidly, and this increase becomes sharper as the network size grows. Our results indicate the presence of a discontinuous transition in the limit of infinitely many nodes. This transition is fundamentally different from the conventional order-to-chaos transition in neural networks, which typically leads to a loss of long-term predictability and a decline in the information processing capacity. Furthermore, we establish a scaling law relating the critical nonlinearity to the noise intensity, which implies that the critical nonlinearity vanishes in the absence of noise.