Reinforcement Learning, Optimal Control, and Bayesian Filtering in Data Assimilation

TL;DR

This paper unifies Bayesian filtering, smoothing, variational data assimilation, and control within a single variational framework, clarifying their relationships and conditions for optimality.

Contribution

It introduces a finite-horizon variational formulation that explicitly connects various data assimilation and control methods, providing new insights into their theoretical foundations.

Findings

Identifies the evidence as a global infimum in the variational hierarchy.

Shows strong- and weak-constraint 4D-Var are MAP estimators under Gaussian assumptions.

Demonstrates the ensemble Kalman filter as a Gaussian approximation in the linear-Gaussian limit.

Abstract

We give a finite-horizon variational formulation that places Bayesian filtering and smoothing, variational data assimilation, KL-regularized control, and Kalman-type methods inside one mathematically explicit hierarchy. For a discrete-time hidden Markov model and any admissible one-step candidate law $q_t$, We prove $J_t(q_t)=\mathbb{E}_{q_t}\!\left[-\log p(y_t\mid X_t)\right] +\mathrm{KL}\!\left(q_t\|p_t^f\right) =\mathrm{KL}\!\left(q_t\|p_t^a\right)-\log p(y_t\mid y_{0:t-1})$, and, for any admissible path law $q$, $J_{\mathrm{path}}(q)=\mathbb{E}_{q}\!\left[-\sum_{t=0}^{T}\log p(y_t\mid X_t)\right] +\mathrm{KL}\!\left(q\|p(x_{0:T})\right) =\mathrm{KL}\!\left(q\|p(x_{0:T}\mid y_{0:T})\right)-\log p(y_{0:T})$. These identities determine the evidence as the global infimum and make the analysis and smoothing posteriors the unique minimizers whenever those posterior laws belong to the admissible classes. This separates targets that are often conflated: strong- and weak-constraint 4D-Var are MAP estimators under the stated Gaussian assumptions; KL-regularized control recovers the Bayesian posterior only when the passive dynamics, likelihood cost, temperature, and a restrictive representability condition on the policy class are all matched correctly; and the linear-Gaussian specialization yields the Kalman analysis exactly. The ensemble Kalman filter then appears as a Gaussian and finite-ensemble approximation to the forecast-to-analysis map, exact only in the linear-Gaussian infinite-ensemble limit. This framework also clarifies RMSE-based RL data assimilation: such rewards may define effective estimators or pseudo-posteriors, but not exact posterior recovery unless they realize the likelihood-plus-KL objective.