Can Transformers predict system collapse in dynamical systems?

TL;DR

This study evaluates whether Transformer models can predict catastrophic system collapse in nonlinear dynamical systems, finding they fail to generalize to collapse scenarios unlike reservoir computing.

Contribution

The paper provides a benchmark comparison showing Transformers' inability to extrapolate system collapse, highlighting limitations in their generalization for dynamical systems.

Findings

Transformers fail to predict bifurcation-induced collapse in unseen parameter regimes.

Reservoir computing reliably predicts system transitions across parameter regimes.

Transformers' permutation-invariant attention may limit their ability to model parameter-dependent dynamics.

Abstract

Transformer architectures have recently surged as promising solutions for nonlinear dynamical systems, proposed as foundation models capable of zero-shot dynamics reconstruction and forecasting. Despite this success, it remains unclear whether they can truly serve as reliable digital twins of dynamical systems, i.e., whether they capture the underlying physical dynamics in distinct parameter regimes, especially in parameter regimes from which no training data is taken. For parameter-space extrapolation in nonlinear dynamical systems, reservoir computing has demonstrated broad success, as proper training can turn it into an intrinsic dynamical system capable of capturing not only the dynamical climate of the target system but more importantly, how the climate changes with parameter. Transformers, in contrast, rely on permutation-invariant attention mechanisms that can limit their ability to capture how temporal structure changes with parameter. To determine if Transformers have the capability of dynamics extrapolation, we take predicting catastrophic collapse, which occurs when a bifurcation parameter crosses a critical threshold, as a benchmark task. Models are trained on trajectories in normal parameter regimes and then tested on parameters in an unseen regime with system collapse. Our results show that Transformers, across configurations, consistently fail to capture collapse, while reservoir computing reliably predicts the transitions. This surprising finding raises questions about the generalization ability of Transformers to dynamical systems, a topic warranting future research.