Out-of-Domain Generalization in Dynamical Systems Reconstruction

TL;DR

This paper analyzes the challenge of out-of-domain generalization in dynamical systems reconstruction, providing a formal framework, theoretical proofs, and empirical evidence showing current deep learning methods often fail to generalize across different domains.

Contribution

It introduces a formal topological and ergodic theory-based framework for understanding out-of-domain generalization in dynamical systems reconstruction and proves the limitations of black-box deep learning methods.

Findings

Deep learning methods often fail to generalize out-of-domain in DSR.

Theoretical proofs show inherent limitations of black-box models without structural priors.

Empirical results confirm the failure of current DSR algorithms to cover the entire phase space.

Abstract

In science we are interested in finding the governing equations, the dynamical rules, underlying empirical phenomena. While traditionally scientific models are derived through cycles of human insight and experimentation, recently deep learning (DL) techniques have been advanced to reconstruct dynamical systems (DS) directly from time series data. State-of-the-art dynamical systems reconstruction (DSR) methods show promise in capturing invariant and long-term properties of observed DS, but their ability to generalize to unobserved domains remains an open challenge. Yet, this is a crucial property we would expect from any viable scientific theory. In this work, we provide a formal framework that addresses generalization in DSR. We explain why and how out-of-domain (OOD) generalization (OODG) in DSR profoundly differs from OODG considered elsewhere in machine learning. We introduce mathematical notions based on topological concepts and ergodic theory to formalize the idea of learnability of a DSR model. We formally prove that black-box DL techniques, without adequate structural priors, generally will not be able to learn a generalizing DSR model. We also show this empirically, considering major classes of DSR algorithms proposed so far, and illustrate where and why they fail to generalize across the whole phase space. Our study provides the first comprehensive mathematical treatment of OODG in DSR, and gives a deeper conceptual understanding of where the fundamental problems in OODG lie and how they could possibly be addressed in practice.