Machine Learning for Predicting Chaotic Systems

TL;DR

This study evaluates various machine learning models for predicting chaotic systems, revealing that simple, well-tuned methods often outperform complex deep learning models, emphasizing the importance of model-data alignment.

Contribution

The paper provides a comprehensive comparison of ML architectures on benchmark datasets, introduces a new uncertainty quantification database and a novel error metric for chaotic systems.

Findings

Simple methods can outperform deep learning models when properly tuned.

Performance varies significantly across different experimental setups.

Introducing the cumulative maximum error metric improves evaluation for chaotic predictions.

Abstract

Predicting chaotic dynamical systems is critical in many scientific fields, such as weather forecasting, but challenging due to the characteristic sensitive dependence on initial conditions. Traditional modeling approaches require extensive domain knowledge, often leading to a shift towards data-driven methods using machine learning. However, existing research provides inconclusive results on which machine learning methods are best suited for predicting chaotic systems. In this paper, we compare different lightweight and heavyweight machine learning architectures using extensive existing benchmark databases, as well as a newly introduced database that allows for uncertainty quantification in the benchmark results. In addition to state-of-the-art methods from the literature, we also present new advantageous variants of established methods. Hyperparameter tuning is adjusted based on computational cost, with more tuning allocated to less costly methods. Furthermore, we introduce the cumulative maximum error, a novel metric that combines desirable properties of traditional metrics and is tailored for chaotic systems. Our results show that well-tuned simple methods, as well as untuned baseline methods, often outperform state-of-the-art deep learning models, but their performance can vary significantly with different experimental setups. These findings highlight the importance of aligning prediction methods with data characteristics and caution against the indiscriminate use of overly complex models.