Policy Optimization in Control: Geometry and Algorithmic Implications

TL;DR

This survey examines the geometric structure of policy optimization in control systems, analyzing how geometry influences stability, performance, and algorithmic design in key control problems like LQR and $\ ext{H}_\infty$ control.

Contribution

It provides a geometric framework for understanding policy parameterization and optimization in control, highlighting the impact on stability and performance of local search algorithms.

Findings

Topology and Riemannian geometry of stabilizing policies analyzed

Structural properties of performance measures explored

Implications for policy optimization algorithms discussed

Abstract

This survey explores the geometric perspective on policy optimization within the realm of feedback control systems, emphasizing the intrinsic relationship between control design and optimization. By adopting a geometric viewpoint, we aim to provide a nuanced understanding of how various ``complete parameterization'' -- referring to the policy parameters together with its Riemannian geometry -- of control design problems, influence stability and performance of local search algorithms. The paper is structured to address key themes such as policy parameterization, the topology and geometry of stabilizing policies, and their implications for various (non-convex) dynamic performance measures. We focus on a few iconic control design problems, including the Linear Quadratic Regulator (LQR), Linear Quadratic Gaussian (LQG) control, and $\mathcal{H}_\infty$ control. In particular, we first discuss the topology and Riemannian geometry of stabilizing policies, distinguishing between their static and dynamic realizations. Expanding on this geometric perspective, we then explore structural properties of the aforementioned performance measures and their interplay with the geometry of stabilizing policies in presence of policy constraints; along the way, we address issues such as spurious stationary points, symmetries of dynamic feedback policies, and (non-)smoothness of the corresponding performance measures. We conclude the survey with algorithmic implications of policy optimization in feedback design.