Panda: A pretrained forecast model for chaotic dynamics

TL;DR

Panda is a pretrained model that accurately forecasts chaotic systems, demonstrating emergent properties and the ability to generalize to unseen systems and real-world data, inspired by dynamical systems theory.

Contribution

We introduce Panda, a novel pretrained model trained on synthetic chaotic data, capable of zero-shot forecasting and understanding complex nonlinear dynamics.

Findings

Panda achieves accurate zero-shot forecasting of unseen chaotic systems.

Panda exhibits nonlinear resonance patterns in attention heads.

Panda can predict partial differential equations without retraining.

Abstract

Chaotic systems are intrinsically sensitive to small errors, challenging efforts to construct predictive data-driven models of real-world dynamical systems such as fluid flows or neuronal activity. Prior efforts comprise either specialized models trained on individual time series, or foundation models trained on vast time series databases with little underlying dynamical structure. Motivated by dynamical systems theory, we present Panda, Patched Attention for Nonlinear DynAmics. We train Panda on a novel synthetic, extensible dataset of $2 \times 10^4$ chaotic dynamical systems that we discover using an evolutionary algorithm. Trained purely on simulated data, Panda exhibits emergent properties: zero-shot forecasting of unseen chaotic systems preserving both short-term accuracy and distributional measures, nonlinear resonance patterns in attention heads, and effective prediction of real-world experimental time series. Despite having been trained only on low-dimensional ordinary differential equations, Panda spontaneously develops the ability to predict partial differential equations without retraining. We also demonstrate a neural scaling law for differential equations, underscoring the potential of pre-trained models for probing abstract mathematical domains like nonlinear dynamics.