Data-Driven Performance Measures using Global Properties of Attractors for Black-Box Surrogate Models of Chaotic Systems

TL;DR

This paper introduces four global property-based, data-driven measures to evaluate and optimize surrogate models of chaotic systems, improving robustness and enabling systematic model ranking and hyperparameter tuning.

Contribution

It presents a novel set of measures based on attractor properties that do not require manual fitting, facilitating robust evaluation and optimization of chaotic system surrogates.

Findings

Measures are robust against initial conditions.

They enable systematic rejection of poor models.

Application improves reservoir computing performance.

Abstract

In climate systems, physiological models, optics, and many more, surrogate models are developed to reconstruct chaotic dynamical systems. We introduce four data-driven measures using global attractor properties to evaluate the quality of the reconstruction of a given surrogate time series. The measures are robust against the initial position of the chaotic system as they are based on empirical approximations of the correlation integral and the probability density function, both of which are global properties of the attractor. In contrast to previous methods, we do not need a manual fitting procedure, making the measures straightforward to evaluate. Using a hypothesis testing framework, we can systematically find and reject surrogate models whose reconstructions significantly differ from the true system. Further, we show that the measures can be used as a statistical ranking metric, which in practice allows for hyperparameter optimization. Applying our measures to reservoir computing with a low number of nodes, we demonstrate how the measures can be used to reject poor reconstructions and use them as a tool to find an optimal spectral radius for which considerably fewer solutions are rejected and the overall quality of the solution improves.