Robust Optimization and Validation of Echo State Networks for learning chaotic dynamics

TL;DR

This paper enhances the robustness of Echo State Networks for predicting chaotic dynamics by proposing new validation strategies and comparing hyperparameter optimization methods, demonstrating improved accuracy and reliability in chaotic system forecasting.

Contribution

It introduces chaos-specific validation strategies and compares Bayesian optimization with grid search, improving ESN robustness for chaotic time series prediction.

Findings

Proposed chaos-aware validation strategies outperform existing methods.

Bayesian optimization yields better hyperparameters than grid search.

Validated on nonlinear systems with chaotic and quasiperiodic solutions.

Abstract

An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The aim of this paper is to assess and improve the robustness of Echo State Networks for the time-accurate prediction of chaotic solutions. The goal is three-fold. First, we investigate the robustness of routinely used validation strategies. Second, we propose the Recycle Validation, and the chaotic versions of existing validation strategies, to specifically tackle the forecasting of chaotic systems. Third, we compare Bayesian optimization with the traditional Grid Search for optimal hyperparameter selection. Numerical tests are performed on two prototypical nonlinear systems that have both chaotic and quasiperiodic solutions. Both model-free and model-informed Echo State Networks are analysed. By comparing the network's robustness in learning chaotic versus quasiperiodic solutions, we highlight fundamental challenges in learning chaotic solutions. The proposed validation strategies, which are based on the dynamical systems properties of chaotic time series, are shown to outperform the state-of-the-art validation strategies. Because the strategies are principled-they are based on chaos theory such as the Lyapunov time-they can be applied to other Recurrent Neural Networks architectures with little modification. This work opens up new possibilities for the robust design and application of Echo State Networks, and Recurrent Neural Networks, to the time-accurate prediction of chaotic systems.