Reinforcement learning-based estimation for partial differential equations

TL;DR

This paper introduces RL-ROE, a reinforcement learning-based reduced-order estimator that improves state estimation accuracy in nonlinear PDE systems like fluid flows, especially with limited sensors.

Contribution

The paper presents a novel RL-based nonlinear policy for ROM correction, outperforming Kalman filters in PDE state estimation tasks.

Findings

RL-ROE outperforms Kalman filter with few sensors.

RL-ROE provides accurate estimates across different physical parameters.

The method effectively compensates ROM errors using reinforcement learning.

Abstract

In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original high-dimensional dynamics onto a computationally tractable low-dimensional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlinearity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier-Stokes equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman filter designed using the same ROM. Moreover, it yields accurate high-dimensional state estimates for trajectories corresponding to various physical parameter values, without direct knowledge of the latter.