Asymptotic Stability in Reservoir Computing

TL;DR

This paper investigates the stability of Reservoir Computing networks using the recurrent kernel limit, providing insights into the stability-chaos boundary and aiding hyperparameter tuning for better performance.

Contribution

It introduces a quantitative analysis of stability in reservoir computing via the recurrent kernel limit, applicable to reservoirs with hundreds of neurons.

Findings

Characterizes the stability-chaos frontier in reservoir computing.

Provides a quantitative tool for hyperparameter tuning.

Enhances understanding of RNN dynamics in large reservoirs.

Abstract

Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel limit to draw new insights on stability in reservoir computing. This limit corresponds to large reservoir sizes, and it already becomes relevant for reservoirs with a few hundred neurons. We obtain a quantitative characterization of the frontier between stability and chaos, which can greatly benefit hyperparameter tuning. In a broader sense, our results contribute to understanding the complex dynamics of Recurrent Neural Networks.