Model scale versus domain knowledge in statistical forecasting of chaotic systems

TL;DR

This study compares 24 forecasting methods on 135 chaotic systems, revealing that large-scale, domain-agnostic models excel in long-term predictions, while physics-based methods perform better with limited data.

Contribution

It provides the largest comparative analysis of forecasting methods on chaotic systems, highlighting the strengths of large-scale models in long-horizon forecasting and the advantages of hybrid methods in data-limited scenarios.

Findings

Large-scale models predict accurately up to two dozen Lyapunov times.

Forecast accuracy in long-horizon regimes decouples from classical invariants like Lyapunov exponents.

Physics-based hybrid methods outperform in data-limited, short-term forecasting.

Abstract

Chaos and unpredictability are traditionally synonymous, yet large-scale machine learning methods recently have demonstrated a surprising ability to forecast chaotic systems well beyond typical predictability horizons. However, recent works disagree on whether specialized methods grounded in dynamical systems theory, such as reservoir computers or neural ordinary differential equations, outperform general-purpose large-scale learning methods such as transformers or recurrent neural networks. These prior studies perform comparisons on few individually-chosen chaotic systems, thereby precluding robust quantification of how statistical modeling choices and dynamical invariants of different chaotic systems jointly determine empirical predictability. Here, we perform the largest to-date comparative study of forecasting methods on the classical problem of forecasting chaos: we benchmark 24 state-of-the-art forecasting methods on a crowdsourced database of 135 low-dimensional systems with 17 forecast metrics. We find that large-scale, domain-agnostic forecasting methods consistently produce predictions that remain accurate up to two dozen Lyapunov times, thereby accessing a new long-horizon forecasting regime well beyond classical methods. We find that, in this regime, accuracy decorrelates with classical invariant measures of predictability like the Lyapunov exponent. However, in data-limited settings outside the long-horizon regime, we find that physics-based hybrid methods retain a comparative advantage due to their strong inductive biases.