When are dynamical systems learned from time series data statistically accurate?

TL;DR

This paper introduces an ergodic theoretic framework to assess when learned neural network models of dynamical systems accurately reproduce their statistical and physical properties, beyond just low test error.

Contribution

It provides a theoretical foundation for understanding the statistical generalization of neural dynamical models, especially in chaotic systems, and explains the benefits of Jacobian information during training.

Findings

Regression methods for dynamical generators often fail to generalize statistically.

Adding Jacobian information during training improves statistical accuracy.

Theoretical analysis verified on various neural architectures and chaotic systems.

Abstract

Conventional notions of generalization often fail to describe the ability of learned models to capture meaningful information from dynamical data. A neural network that learns complex dynamics with a small test error may still fail to reproduce its \emph{physical} behavior, including associated statistical moments and Lyapunov exponents. To address this gap, we propose an ergodic theoretic approach to generalization of complex dynamical models learned from time series data. Our main contribution is to define and analyze generalization of a broad suite of neural representations of classes of ergodic systems, including chaotic systems, in a way that captures emulating underlying invariant, physical measures. Our results provide theoretical justification for why regression methods for generators of dynamical systems (Neural ODEs) fail to generalize, and why their statistical accuracy improves upon adding Jacobian information during training. We verify our results on a number of ergodic chaotic systems and neural network parameterizations, including MLPs, ResNets, Fourier Neural layers, and RNNs.