Edge of stability echo state networks

TL;DR

This paper introduces the Edge of Stability Echo State Network (ES2N), a new architecture that balances memory retention and nonlinearity, achieving maximum short-term memory capacity and improved nonlinear modeling performance.

Contribution

The paper proposes the ES2N architecture, combining nonlinear and orthogonal reservoirs, with a mathematical analysis showing its dynamics operate near the edge-of-chaos for optimal memory.

Findings

ES2N reaches maximum short-term memory capacity.

ES2N offers a better trade-off between memory and nonlinearity.

ES2N significantly improves autoregressive nonlinear modeling performance.

Abstract

Echo State Networks (ESNs) are time-series processing models working under the Echo State Property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. With the goal of bringing together the fading memory property and the ability to retain as much memory as possible, in this paper we introduce a new ESN architecture, called the Edge of Stability Echo State Network (ES$^2$N). The introduced ES$^2$N model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. We provide a thorough mathematical analysis of the introduced model, proving that the whole eigenspectrum of the Jacobian of the ES$^2$N map can be contained in an annular neighbourhood of a complex circle of controllable radius, and exploit this property to demonstrate that the ES$^2$N's forward dynamics evolves close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that the newly introduced reservoir model is able to reach the theoretical maximum short-term memory capacity. At the same time, in comparison to standard ESN, ES$^2$N is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling.