On the emergence of numerical instabilities in Next Generation Reservoir Computing

TL;DR

This paper investigates how the numerical conditioning of the feature matrix in Next Generation Reservoir Computing affects its stability and performance, proposing SVD-based training as an effective method to improve accuracy without regularization.

Contribution

It uncovers the link between matrix conditioning and NGRC dynamics, and demonstrates SVD-based training as a regularization-free, accurate approach.

Findings

NGRC feature matrix tends to be ill-conditioned for certain hyperparameters.

SVD-based training achieves accurate forecasts without regularization.

Ill-conditioning is influenced by short time lags, high polynomial degree, and limited data.

Abstract

Next Generation Reservoir Computing (NGRC) is a low-cost machine learning method for forecasting chaotic time series from data. Computational efficiency is crucial for scalable reservoir computing, requiring better strategies to reduce training cost. In this work, we uncover a connection between the numerical conditioning of the NGRC feature matrix -- formed by polynomial evaluations on time-delay coordinates -- and the long-term NGRC dynamics. We show that NGRC can be trained without regularization, reducing computational time. Our contributions are twofold. First, merging tools from numerical linear algebra and ergodic theory of dynamical systems, we systematically study how the feature matrix conditioning varies across hyperparameters. We demonstrate that the NGRC feature matrix tends to be ill-conditioned for short time lags, high-degree polynomials, and short length of training data. Second, we evaluate the impact of different numerical algorithms (Cholesky, singular value decomposition (SVD), and lower-upper (LU) decomposition) for solving the regularized least-squares problem. Our results reveal that SVD-based training achieves accurate forecasts without regularization, being preferable when compared against the other algorithms.