Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games

TL;DR

This paper introduces projection-free policy gradient algorithms for linear-quadratic dynamic games, demonstrating global convergence to Nash equilibrium with different rates depending on the leader's policy update method.

Contribution

It proposes novel, projection-free, sequential algorithms for LQ games with proven global convergence properties, extending to model-free scenarios.

Findings

Natural gradient descent/ascent achieves global sublinear convergence.

Quasi-Newton policies attain global quadratic convergence.

Addresses stabilization and indefinite cost issues in policy updates.

Abstract

We propose projection-free sequential algorithms for linear-quadratic dynamics games. These policy gradient based algorithms are akin to Stackelberg leadership model and can be extended to model-free settings. We show that if the leader performs natural gradient descent/ascent, then the proposed algorithm has a global sublinear convergence to the Nash equilibrium. Moreover, if the leader adopts a quasi-Newton policy, the algorithm enjoys a global quadratic convergence. Along the way, we examine and clarify the intricacies of adopting sequential policy updates for LQ games, namely, issues pertaining to stabilization, indefinite cost structure, and circumventing projection steps.