Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

TL;DR

This paper proves the first efficient convergence guarantee for a primal-dual actor-critic algorithm with nonlinear function approximation in RL, demonstrating its effectiveness through theoretical analysis and empirical validation.

Contribution

It introduces a novel convergence analysis for primal-dual actor-critic with nonlinear approximation, applicable under broad RL scenarios including multi-agent settings.

Findings

Convergence rate of 0(rac{\

Empirical results on OpenAI Gym tasks support theoretical claims.

Applicable to various RL settings, including multi-agent RL.

Abstract

We study the convergence of the actor-critic algorithm with nonlinear function approximation under a nonconvex-nonconcave primal-dual formulation. Stochastic gradient descent ascent is applied with an adaptive proximal term for robust learning rates. We show the first efficient convergence result with primal-dual actor-critic with a convergence rate of $\mathcal{O}\left(\sqrt{\frac{\ln \left(N d G^2 \right)}{N}}\right)$ under Markovian sampling, where $G$ is the element-wise maximum of the gradient, $N$ is the number of iterations, and $d$ is the dimension of the gradient. Our result is presented with only the Polyak-\L{}ojasiewicz condition for the dual variables, which is easy to verify and applicable to a wide range of reinforcement learning (RL) scenarios. The algorithm and analysis are general enough to be applied to other RL settings, like multi-agent RL. Empirical results on OpenAI Gym continuous control tasks corroborate our theoretical findings.