Reinforcement Learning in Linear Quadratic Deep Structured Teams: Global Convergence of Policy Gradient Methods

TL;DR

This paper proves that policy gradient methods can globally converge to optimal solutions in linear quadratic deep structured teams, with scalable algorithms for large populations and simulations confirming the theoretical findings.

Contribution

It introduces scalable policy gradient algorithms for deep structured teams and establishes their global convergence under certain conditions.

Findings

Model-based policy gradient methods converge globally for small risk factors or large populations.

Model-free and natural policy gradient algorithms are developed for risk-neutral cases.

Simulations verify the theoretical convergence results.

Abstract

In this paper, we study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms for linear quadratic deep structured teams. In such systems, agents are partitioned into a few sub-populations wherein the agents in each sub-population are coupled in the dynamics and cost function through a set of linear regressions of the states and actions of all agents. Every agent observes its local state and the linear regressions of states, called deep states. For a sufficiently small risk factor and/or sufficiently large population, we prove that model-based policy gradient methods globally converge to the optimal solution. Given an arbitrary number of agents, we develop model-free policy gradient and natural policy gradient algorithms for the special case of risk-neutral cost function. The proposed algorithms are scalable with respect to the number of agents due to the fact that the dimension of their policy space is independent of the number of agents in each sub-population. Simulations are provided to verify the theoretical results.