LQR through the Lens of First Order Methods: Discrete-time Case

TL;DR

This paper analyzes the LQR problem using first order optimization methods, demonstrating smoothness, coercivity, and exponential stability of flows, and providing convergence guarantees for discretized algorithms.

Contribution

It introduces a first order optimization perspective to LQR, analyzing gradient, natural gradient, and quasi-Newton flows, with convergence guarantees and stepsize criteria.

Findings

Flows are exponentially stable and admit unique solutions.

Gradient descent and natural gradient descent converge linearly with proper stepsizes.

Quasi-Newton iteration achieves quadratic convergence, recovering the Hewer algorithm.

Abstract

We consider the Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. Such a setup facilitates examining the implications of a natural initial-state independent formulation of LQR in designing first order algorithms. It is shown that this cost function is smooth and coercive, and provide an alternate means of noting its gradient dominated property. In the process, we provide a number of analytic observations on the LQR cost when directly analyzed in terms of the feedback gain. We then examine three types of well-posed flows for LQR: gradient flow, natural gradient flow and the quasi-Newton flow. The coercive property suggests that these flows admit unique solutions while gradient dominated property indicates that the corresponding Lyapunov functionals decay at an exponential rate; we also prove that these flows are exponentially stable in the sense of Lyapunov. We then discuss the forward Euler discretization of these flows, realized as gradient descent, natural gradient descent and the quasi-Newton iteration. We present stepsize criteria for gradient descent and natural gradient descent, guaranteeing that both algorithms converge linearly to the global optima. An optimal stepsize for the quasi-Newton iteration is also proposed, guaranteeing a $Q$-quadratic convergence rate--and in the meantime--recovering the Hewer algorithm.