Convergence Analysis of Gradient Flow for Overparameterized LQR Formulations

TL;DR

This paper provides a theoretical analysis of gradient flow convergence in overparameterized neural network-based LQR control, demonstrating conditions for optimality, stability, and accelerated convergence through both proofs and numerical experiments.

Contribution

It introduces a novel convergence analysis for overparameterized neural network controllers in LQR problems, including proofs of optimality and stability, and shows empirical acceleration in training.

Findings

Conservation law of the system's gradient structure.

Global convergence to optimal feedback for single hidden layer networks.

Empirical evidence of faster convergence with overparameterization.

Abstract

Motivated by the growing use of artificial intelligence (AI) tools in control design, this paper analyses the intersection between results from gradient methods for the model-free linear quadratic regulator (LQR), and linear feedforward neural networks (LFFNNs), More specifically, it looks into the case where one wants to find a LFFNN feedback that minimizes a LQR cost. This paper starts by analyzing the structure of the gradient expression for the parameters of each layer, which implies a key conservation law of the system. This conservation law is then leveraged to generalize existing results on boundedness and global convergence of solutions to critical points, and invariance of the set of stabilizing networks under the training dynamics. This is followed by an analysis of the case where the LFFNN has a single hidden layer, for which the paper proves that the training converges not only to the critical points, but to the optimal feedback control law for all but a set of Lebesgue measure zero of the initializations. These theoretical results are followed by an extensive analysis of a simple version of the problem -- the ``vector case'' -- proving the theoretical properties of accelerated convergence and small-input input-to-state stability (ISS) for this simpler example. Finally, the paper presents numerical evidence of faster convergence of the training of general LFFNNs when compared to non-overparameterized formulations, showing that the acceleration of the solution is observable even when the gradient is not explicitly computed, but estimated from evaluations of the cost function.