On the Gradient Domination of the LQG Problem

TL;DR

This paper investigates policy gradient methods for the LQG problem, introducing a novel history-based parameterization that establishes gradient dominance and guarantees global convergence.

Contribution

It proposes a new history representation parameterization for LQG, enabling the first theoretical guarantees of gradient dominance and convergence for policy gradient methods in this setting.

Findings

Establishes gradient dominance for the LQG cost with the new parameterization.

Proves global convergence and stability guarantees for policy gradient LQG.

Numerical experiments confirm theoretical results and illustrate convergence behavior.

Abstract

We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study PG for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. This representation enables us to establish gradient dominance and approximate smoothness for the LQG cost. We prove global convergence and per-iteration stability guarantees for policy gradient LQG in model-based and model-free settings. Numerical experiments on an open-loop unstable system are provided to support the global convergence guarantees and to illustrate convergence under different history lengths of the history representation.