Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation

TL;DR

This paper introduces a machine learning-enhanced conditional mean filter that generalizes the ensemble Kalman filter for nonlinear data assimilation, improving accuracy and stability in complex systems.

Contribution

It develops a systematic methodology integrating machine learning into the conditional mean filter, enhancing its performance and robustness over traditional methods.

Findings

ML-EnCMF outperforms EnKF and likelihood-based EnCMF in tests

The method effectively reduces statistical errors in small ensemble sizes

Demonstrated success on Lorenz-63 and Lorenz-96 systems

Abstract

This paper presents the machine learning-based ensemble conditional mean filter (ML-EnCMF) -- a filtering method based on the conditional mean filter (CMF) previously introduced in the literature. The updated mean of the CMF matches that of the posterior, obtained by applying Bayes' rule on the filter's forecast distribution. Moreover, we show that the CMF's updated covariance coincides with the expected conditional covariance. Implementing the EnCMF requires computing the conditional mean (CM). A likelihood-based estimator is prone to significant errors for small ensemble sizes, causing the filter divergence. We develop a systematical methodology for integrating machine learning into the EnCMF based on the CM's orthogonal projection property. First, we use a combination of an artificial neural network (ANN) and a linear function, obtained based on the ensemble Kalman filter (EnKF), to approximate the CM, enabling the ML-EnCMF to inherit EnKF's advantages. Secondly, we apply a suitable variance reduction technique to reduce statistical errors when estimating loss function. Lastly, we propose a model selection procedure for element-wisely selecting the applied filter, i.e., either the EnKF or ML-EnCMF, at each updating step. We demonstrate the ML-EnCMF performance using the Lorenz-63 and Lorenz-96 systems and show that the ML-EnCMF outperforms the EnKF and the likelihood-based EnCMF.