Global Convergence of Policy Gradient Methods for ReLU Controllers in Linear Quadratic Regulation

TL;DR

This paper proves that policy gradient methods with overparameterized ReLU neural controllers reliably converge to the optimal solution in scalar linear quadratic regulation, despite nonconvexities.

Contribution

It provides the first rigorous analysis of policy gradient convergence for overparameterized ReLU controllers in LQR, revealing stability and geometric convergence.

Findings

Policy gradient converges to the optimal LQR controller with high probability.

Overparameterized ReLU controllers can be analyzed via effective gains on positive and negative half-lines.

Convergence is guaranteed under suitable initialization, width, and step size.

Abstract

We study the convergence of model-based policy gradient for the deterministic, scalar, discounted linear-quadratic regulator when the controller is an overparameterized one-hidden-layer ReLU network without biases. Although the optimal LQR controller is linear, neural parameterization creates a redundant nonconvex weight space with a possibly asymmetric piecewise-linear controller. We show that this structure can still be analyzed exactly through the two effective gains induced on the positive and negative half-lines. Under suitable random initialization, sufficient width, and a small step size, the model-based policy gradient remains stable, decreases the cost geometrically, and drives the effective gains to the unique optimal scalar LQR gain with high probability.