A Data-Consistent Approach to Ensemble Filtering

TL;DR

This paper introduces QPCA-EnDCF, a deterministic ensemble filtering method that improves accuracy and calibration in chaotic systems by spectral regularization and rank-restricted updates, outperforming stochastic EnKF variants.

Contribution

The paper develops a data-consistent, deterministic ensemble filter using spectral regularization, providing a theoretical bias-variance analysis and demonstrating improved performance over stochastic EnKF in experiments.

Findings

QPCA-EnDCF reduces spread-error mismatch and improves reliability.

The method achieves lower RMSE in Lorenz-96 system experiments.

Stochastic EnKF has an irreducible variance term, while QPCA-EnDCF's variability depends on eigenspace stability.

Abstract

Ensemble filtering of chaotic, partially observed systems is often performed with ensembles far smaller than the state dimension resulting in empirical covariances that are low rank. Subsequently, stochastic observation perturbations can degrade both accuracy and probabilistic calibration. We develop a data-consistent perspective on ensemble filtering and introduce the Quantity-of-Interest Principal Component Analysis Ensemble Data Consistent Filter (QPCA-EnDCF), which is a deterministic method that replaces perturbed observations with a spectrally regularized update in observation space. The method whitens forecast--observation residuals, computes an empirical eigendecomposition of the residual covariance, and restricts the correction to a rank-$\kappa$ subspace before mapping the increment back to state space through an empirical gain. We establish a theoretical framework that separates population and finite-ensemble objects and yields a bias--variance decomposition for the analysis mean. The analysis shows that stochastic EnKF variants incur an irreducible $\mathcal{O}(1/N)$ variance contribution from observation perturbations, whereas QPCA-EnDCF replaces this term with projector-estimation variability that is also $\mathcal{O}(1/N)$ but depends on the retained rank and the cutoff gap through eigenspace stability. Numerical experiments on the Lorenz--96 system in strongly undersampled regimes demonstrate that QPCA-EnDCF substantially improves spread--skill behavior, temporal tracking between spread and error, and rank-histogram reliability relative to sequential and four-dimensional stochastic EnKF. Under the baseline configuration, these calibration gains are accompanied by lower RMSE.