LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations

TL;DR

LD-EnSF introduces a fast, efficient data assimilation method that operates in a learned latent space, significantly reducing computational costs while accurately handling sparse, noisy observations in high-dimensional systems.

Contribution

The paper presents LD-EnSF, a novel latent space score-based data assimilation approach that eliminates full-space simulations and improves efficiency with surrogate dynamics and history-aware encoding.

Findings

Achieves orders-of-magnitude speedup over existing methods

Maintains high accuracy and robustness with sparse, noisy data

Effective on high-dimensional benchmarks

Abstract

Data assimilation techniques are crucial for accurately tracking complex dynamical systems by integrating observational data with numerical forecasts. Recently, score-based data assimilation methods emerged as powerful tools for high-dimensional and nonlinear data assimilation. However, these methods still incur substantial computational costs due to the need for expensive forward simulations. In this work, we propose LD-EnSF, a novel score-based data assimilation method that eliminates the need for full-space simulations by evolving dynamics directly in a compact latent space. Our method incorporates improved Latent Dynamics Networks (LDNets) to learn accurate surrogate dynamics and introduces a history-aware LSTM encoder to effectively process sparse and irregular observations. By operating entirely in the latent space, LD-EnSF achieves speedups orders of magnitude over existing methods while maintaining high accuracy and robustness. We demonstrate the effectiveness of LD-EnSF on several challenging high-dimensional benchmarks with highly sparse (in both space and time) and noisy observations.