Ensemble Kalman filter in latent space using a variational autoencoder pair

TL;DR

This paper introduces a hybrid data assimilation method combining ensemble Kalman filtering with variational autoencoders in latent space, improving handling of non-Gaussian and constrained variables in dynamic systems.

Contribution

The authors propose a novel ensemble Kalman filter variant that incorporates VAEs in latent space, enabling better adherence to data manifolds and robustness against non-Gaussian errors.

Findings

VAE ensures ensemble members stay close to the true data manifold.

Online updating of the VAE is feasible for non-stationary manifolds.

Using a second latent space improves robustness to non-Gaussian observational errors.

Abstract

Popular (ensemble) Kalman filter data assimilation (DA) approaches assume that the errors in both the a priori estimate of the state and those in the observations are Gaussian. For constrained variables, e.g. sea ice concentration or stress, such an assumption does not hold. The variational autoencoder (VAE) is a machine learning (ML) technique that allows to map an arbitrary distribution to/from a latent space in which the distribution is supposedly closer to a Gaussian. We propose a novel hybrid DA-ML approach in which VAEs are incorporated in the DA procedure. Specifically, we introduce a variant of the popular ensemble transform Kalman filter (ETKF) in which the analysis is applied in the latent space of a single VAE or a pair of VAEs. In twin experiments with a simple circular model, whereby the circle represents an underlying submanifold to be respected, we find that the use of a VAE ensures that a posteri ensemble members lie close to the manifold containing the truth. Furthermore, online updating of the VAE is necessary and achievable when this manifold varies in time, i.e. when it is non-stationary. We demonstrate that introducing an additional second latent space for the observational innovations improves robustness against detrimental effects of non-Gaussianity and bias in the observational errors but it slightly lessens the performance if observational errors are strictly Gaussian.