Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality

TL;DR

This paper demonstrates that in certain optimal and robust control problems with partial observability, all Clarke stationary points are globally optimal despite nonconvexity, using a new Extended Convex Lifting framework.

Contribution

It introduces the ECL framework that unifies nonconvex policy optimization with convex reformulations, establishing global optimality of stationary points in complex control problems.

Findings

All Clarke stationary points are globally optimal in the studied control problems.

No spurious local minima exist for the considered nonconvex control problems.

The ECL framework provides a new tool for analyzing nonconvex control landscapes.

Abstract

Direct policy search has achieved great empirical success in reinforcement learning. Many recent studies have revisited its theoretical foundation for continuous control, which reveals elegant nonconvex geometry in various benchmark problems, especially in fully observable state-feedback cases. This paper considers two fundamental optimal and robust control problems with partial observability: the Linear Quadratic Gaussian (LQG) control with stochastic noises, and $\mathcal{H}_\infty$ robust control with adversarial noises. In the policy space, the former problem is smooth but nonconvex, while the latter one is nonsmooth and nonconvex. We highlight some interesting and surprising ``discontinuity'' of LQG and $\mathcal{H}_\infty$ cost functions around the boundary of their domains. Despite the lack of convexity (and possibly smoothness), our main results show that for a class of non-degenerate policies, all Clarke stationary points are globally optimal and there is no spurious local minimum for both LQG and $\mathcal{H}_\infty$ control. Our proof techniques rely on a new and unified framework of Extended Convex Lifting (ECL), which reconciles the gap between nonconvex policy optimization and convex reformulations. This ECL framework is of independent interest, and we will discuss its details in Part II of this paper.