Neural Policy Gradient Methods: Global Optimality and Rates of Convergence

TL;DR

This paper proves that neural policy gradient methods in overparameterized settings converge to globally optimal policies with sublinear rates, providing the first such guarantees for these methods.

Contribution

It establishes the first global optimality and convergence guarantees for neural policy gradient methods in the overparameterized regime.

Findings

Neural natural policy gradient converges to a global optimum at a sublinear rate.

Neural vanilla policy gradient converges sublinearly to a stationary point.

Global optimality of stationary points is linked to the neural network's representation power.

Abstract

Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at all. We answer both the questions affirmatively in the overparameterized regime. In detail, we prove that neural natural policy gradient converges to a globally optimal policy at a sublinear rate. Also, we show that neural vanilla policy gradient converges sublinearly to a stationary point. Meanwhile, by relating the suboptimality of the stationary points to the representation power of neural actor and critic classes, we prove the global optimality of all stationary points under mild regularity conditions. Particularly, we show that a key to the global optimality and convergence is the "compatibility" between the actor and critic, which is ensured by sharing neural architectures and random initializations across the actor and critic. To the best of our knowledge, our analysis establishes the first global optimality and convergence guarantees for neural policy gradient methods.