Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games

TL;DR

This paper proves that the natural policy gradient method converges to the Nash equilibrium in stochastic N-player linear-quadratic games, with noise being essential for convergence, supported by theoretical analysis and numerical experiments.

Contribution

It establishes the global convergence of policy gradient methods in stochastic linear-quadratic games and identifies noise as a key factor for convergence.

Findings

Policy gradient converges to Nash equilibrium with sufficient noise.

Noise lower bound depends on model parameters.

Numerical experiments confirm noise-induced convergence.

Abstract

We consider a general-sum N-player linear-quadratic game with stochastic dynamics over a finite horizon and prove the global convergence of the natural policy gradient method to the Nash equilibrium. In order to prove the convergence of the method, we require a certain amount of noise in the system. We give a condition, essentially a lower bound on the covariance of the noise in terms of the model parameters, in order to guarantee convergence. We illustrate our results with numerical experiments to show that even in situations where the policy gradient method may not converge in the deterministic setting, the addition of noise leads to convergence.