Reinforcement learning based data assimilation for unknown state model

TL;DR

This paper introduces a reinforcement learning-based data assimilation method that learns surrogate state transition models directly from noisy observations without true state trajectories, improving accuracy in unknown and high-dimensional systems.

Contribution

It proposes a novel reinforcement learning framework integrated with Bayesian filtering to learn surrogate models from noisy data without ground-truth states.

Findings

Achieves superior accuracy in high-dimensional systems

Demonstrates robustness across various observation scenarios

Outperforms existing methods in numerical experiments

Abstract

Data assimilation (DA) has increasingly emerged as a critical tool for state estimation across a wide range of applications. It is significantly challenging when the governing equations of the underlying dynamics are unknown. To this end, various machine learning approaches have been employed to construct a surrogate state transition model in a supervised learning framework, which relies on pre-computed training datasets. However, it is often infeasible to obtain noise-free ground-truth state sequences in practice. To address this challenge, we propose a novel method that integrates reinforcement learning with ensemble-based Bayesian filtering methods, enabling the learning of surrogate state transition model for unknown dynamics directly from noisy observations, without using true state trajectories. Specifically, we treat the process for computing maximum likelihood estimation of surrogate model parameters as a sequential decision-making problem, which can be formulated as a discrete-time Markov decision process (MDP). Under this formulation, learning the surrogate transition model is equivalent to finding an optimal policy of the MDP, which can be effectively addressed using reinforcement learning techniques. Once the model is trained offline, state estimation can be performed in the online stage using filtering methods based on the learned dynamics. The proposed framework accommodates a wide range of observation scenarios, including nonlinear and partially observed measurement models. A few numerical examples demonstrate that the proposed method achieves superior accuracy and robustness in high-dimensional settings.