Echo State Condition at the Critical Point

TL;DR

This paper investigates the conditions under which recurrent neural networks with Lipschitz continuous transfer functions at the critical point (largest singular value equals one) still satisfy the echo state property, emphasizing the role of transfer function shape.

Contribution

It provides a proof for echo state networks at the critical point where the largest singular value is one, highlighting the importance of transfer function shape and introducing a mathematical definition for critical echo state networks.

Findings

Networks can be echo state at the critical point with singular value one.

Transfer function shape determines echo state property at criticality.

Examples illustrate effects of critical connectivity on network behavior.

Abstract

Recurrent networks with transfer functions that fulfill the Lipschitz continuity with K=1 may be echo state networks if certain limitations on the recurrent connectivity are applied. It has been shown that it is sufficient if the largest singular value of the recurrent connectivity is smaller than 1. The main achievement of this paper is a proof under which conditions the network is an echo state network even if the largest singular value is one. It turns out that in this critical case the exact shape of the transfer function plays a decisive role in determining whether the network still fulfills the echo state condition. In addition, several examples with one neuron networks are outlined to illustrate effects of critical connectivity. Moreover, within the manuscript a mathematical definition for a critical echo state network is suggested.