A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems

TL;DR

This paper presents a novel data assimilation framework using an ensemble score filter to enhance machine learning models for long-term forecasting of complex dynamical systems, reducing predictive uncertainty.

Contribution

It introduces a hybrid framework combining ML with the Ensemble Score Filter for improved long-term predictions in high-dimensional nonlinear systems.

Findings

Effective reduction of predictive uncertainty in Lorenz-96 system

Improved long-term forecasting accuracy for KdV equation

Demonstrated robustness of EnSF-enhanced ML models

Abstract

We introduce a score-filter-enhanced data assimilation framework designed to reduce predictive uncertainty in machine learning (ML) models for data-driven dynamical system forecasting. Machine learning serves as an efficient numerical model for predicting dynamical systems. However, even with sufficient data, model uncertainty remains and accumulates over time, causing the long-term performance of ML models to deteriorate. To overcome this difficulty, we integrate data assimilation techniques into the training process to iteratively refine the model predictions by incorporating observational information. Specifically, we apply the Ensemble Score Filter (EnSF), a generative AI-based training-free diffusion model approach, for solving the data assimilation problem in high-dimensional nonlinear complex systems. This leads to a hybrid data assimilation-training framework that combines ML with EnSF to improve long-term predictive performance. We shall demonstrate that EnSF-enhanced ML can effectively reduce predictive uncertainty in ML-based Lorenz-96 system prediction and the Korteweg-De Vries (KdV) equation prediction.