Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence

TL;DR

This paper develops a policy gradient primal-dual approach for solving cost-constrained LQR problems, establishing strong duality, convergence guarantees, and validating results through simulations in safety-critical control scenarios.

Contribution

It introduces a novel primal-dual policy gradient method for constrained LQR, proving strong duality and convergence despite non-convexity, with theoretical and empirical validation.

Findings

Strong duality holds for the cost-constrained LQR problem.

The proposed method converges to the optimal solution.

Simulations confirm theoretical convergence and effectiveness.

Abstract

In safety-critical applications, reinforcement learning (RL) needs to consider safety constraints. However, theoretical understandings of constrained RL for continuous control are largely absent. As a case study, this paper presents a cost-constrained LQR formulation, where a number of LQR costs with user-defined penalty matrices are subject to constraints. To solve it, we propose a policy gradient primal-dual method to find an optimal state feedback gain. Despite the non-convexity of the cost-constrained LQR problem, we provide a constructive proof for strong duality and a geometric interpretation of an optimal multiplier set. By proving that the concave dual function is Lipschitz smooth, we further provide convergence guarantees for the PG primal-dual method. Finally, we perform simulations to validate our theoretical findings.