Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective

TL;DR

This paper introduces a frequency domain approach to certify global optimality in the non-convex LQG control problem, extending to infinite-dimensional controller spaces and including a data-driven extension, with theoretical guarantees and numerical validation.

Contribution

It derives a necessary and sufficient optimality condition for stationary points in parameterized LQG problems, and develops a gradient-based algorithm with global convergence in an infinite-dimensional space.

Findings

A tractable optimality certificate based on controllability and observability.

A gradient algorithm with proven global convergence.

Numerical experiments validating theoretical results.

Abstract

The Linear Quadratic Gaussian (LQG) problem is a classic and widely studied model in optimal control, providing a fundamental framework for designing controllers for linear systems subject to process and observation noises. In recent years, researchers have increasingly focused on directly parameterizing dynamic controllers and optimizing the LQG cost over the resulting parameterized set. However, this parameterization typically gives rise to a highly non-convex optimization landscape for the resulting parameterized LQG problem. To our knowledge, there is currently no general method for certifying the global optimality of candidate controller parameters in this setting. In this work, we address these gaps with the following contributions. First, we derive a necessary and sufficient condition for the global optimality of stationary points in a parameterized LQG problems. This condition reduces the verification of optimality to a test of the controllability and observability for a novel, specially constructed transfer function, yielding a precise and computationally tractable certificate. Furthermore, our condition provides a rigorous explanation for why traditional parameterizations can lead to suboptimal stationary points. Second, we elevate the controller parameter space from conventional finite-dimensional settings to the infinite-dimensional $\mathcal{RH}_\infty$ space and develop a gradient-based algorithm in this setting, for which we provide a theoretical analysis establishing global convergence. Finally, representative numerical experiments validate the theoretical findings and demonstrate the practical viability of the proposed approach. Additionally, the appendix section explores a data-driven extension to the model-free setting, where we outline a parameter estimation scheme and demonstrate its practical viability through numerical simulation.