Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designs

TL;DR

This paper introduces the RHPG algorithm, a reinforcement learning method with proven global convergence for designing optimal linear estimators like the Kalman filter, without prior system knowledge.

Contribution

It develops the first PG algorithm with convergence guarantees for Kalman filter design, integrating dynamic programming with policy search to handle non-convexity.

Findings

Proves global convergence of RHPG for Kalman filter design.

Demonstrates RHPG's effectiveness on a large-scale convection-diffusion model.

Provides theoretical analysis of optimization landscape and sample complexity.

Abstract

We introduce the receding-horizon policy gradient (RHPG) algorithm, the first PG algorithm with provable global convergence in learning the optimal linear estimator designs, i.e., the Kalman filter (KF). Notably, the RHPG algorithm does not require any prior knowledge of the system for initialization and does not require the target system to be open-loop stable. The key of RHPG is that we integrate vanilla PG (or any other policy search directions) into a dynamic programming outer loop, which iteratively decomposes the infinite-horizon KF problem that is constrained and non-convex in the policy parameter into a sequence of static estimation problems that are unconstrained and strongly-convex, thus enabling global convergence. We further provide fine-grained analyses of the optimization landscape under RHPG and detail the convergence and sample complexity guarantees of the algorithm. This work serves as an initial attempt to develop reinforcement learning algorithms specifically for control applications with performance guarantees by utilizing classic control theory in both algorithmic design and theoretical analyses. Lastly, we validate our theories by deploying the RHPG algorithm to learn the Kalman filter design of a large-scale convection-diffusion model. We open-source the code repository at \url{https://github.com/xiangyuan-zhang/LearningKF}.