Trimmed Ensemble Kalman Filter for Nonlinear and Non-Gaussian Data Assimilation Problems

TL;DR

This paper introduces a trimmed ensemble Kalman filter (TEnKF) that adaptively balances accuracy and efficiency in nonlinear, non-Gaussian data assimilation by removing outliers, improving convergence and robustness over traditional methods.

Contribution

The paper proposes the TEnKF, a novel algorithm that interpolates between EnKF and particle filters, with adaptive trimming to enhance performance in complex nonlinear systems.

Findings

TEnKF reproduces limiting distributions of EnKF and PF for specific trimming functions.

The adaptive TEnKF improves convergence and robustness in Lorenz models.

Substantial performance gains over traditional EnKF in nonlinear scenarios.

Abstract

We study the ensemble Kalman filter (EnKF) algorithm for sequential data assimilation in a general situation, that is, for nonlinear forecast and measurement models with non-additive and non-Gaussian noises. Such applications traditionally force us to choose between inaccurate Gaussian assumptions that permit efficient algorithms (e.g., EnKF), or more accurate direct sampling methods which scale poorly with dimension (e.g., particle filters, or PF). We introduce a trimmed ensemble Kalman filter (TEnKF) which can interpolate between the limiting distributions of the EnKF and PF to facilitate adaptive control over both accuracy and efficiency. This is achieved by introducing a trimming function that removes non-Gaussian outliers that introduce errors in the correlation between the model and observed forecast, which otherwise prevent the EnKF from proposing accurate forecast updates. We show for specific trimming functions that the TEnKF exactly reproduces the limiting distributions of the EnKF and PF. We also develop an adaptive implementation which provides control of the effective sample size and allows the filter to overcome periods of increased model nonlinearity. This algorithm allow us to demonstrate substantial improvements over the traditional EnKF in convergence and robustness for the nonlinear Lorenz-63 and Lorenz-96 models.