Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control

TL;DR

This paper reviews recent advances in policy gradient methods, emphasizing their global convergence guarantees and finite-time rates in reinforcement learning, games, and control.

Contribution

It provides a comprehensive overview of recent progress in establishing global convergence and finite-time guarantees for policy gradient algorithms.

Findings

Recent progress in global convergence analysis

Finite-time convergence rates established

Applicability to reinforcement learning, games, and control

Abstract

Policy gradient methods, where one searches for the policy of interest by maximizing the value functions using first-order information, become increasingly popular for sequential decision making in reinforcement learning, games, and control. Guaranteeing the global optimality of policy gradient methods, however, is highly nontrivial due to nonconcavity of the value functions. In this exposition, we highlight recent progresses in understanding and developing policy gradient methods with global convergence guarantees, putting an emphasis on their finite-time convergence rates with regard to salient problem parameters.