A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

TL;DR

This paper introduces two non-linear extensions of the Ensemble Kalman Filter, the CG-EnKF and NS-EnKF, which outperform a deep learning-based particle filter in high-dimensional data assimilation tasks.

Contribution

The paper proposes the conditional-Gaussian and normal score EnKF methods, offering a more effective alternative to existing filters in non-linear, high-dimensional data assimilation.

Findings

CG-EnKF and NS-EnKF outperform the score filter in Lorenz-96 system

NS-EnKF generally outperforms CG-EnKF in handling non-Gaussian noise

Both methods effectively manage highly non-Gaussian perturbations

Abstract

Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.