How more data can hurt: Instability and regularization in next-generation reservoir computing

TL;DR

This paper reveals that in data-driven dynamical system models like NGRC, increasing data can cause instability due to ill-conditioning, and proposes regularization strategies to mitigate this issue.

Contribution

It demonstrates the counter-intuitive phenomenon of data-induced instability in NGRC and introduces simple regularization methods to improve stability.

Findings

More data can lead to instability in NGRC models.

Regularization and noise addition can mitigate instability.

Proper regularization is crucial for stable data-driven dynamical modeling.

Abstract

It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems.