Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER

TL;DR

This paper proves the convergence of actor-critic reinforcement learning algorithms, including PPO and RUDDER, with deep neural networks, using two time-scale stochastic approximation theory, under more general conditions than previous proofs.

Contribution

It provides the first convergence proof for actor-critic methods with deep neural networks, episodic samples, and greedy policies, extending prior linear-function approximation results.

Findings

Proves convergence of PPO and RUDDER under common assumptions.

Extends convergence results to deep neural network function approximations.

Handles episodic samples and policies that become greedy during learning.

Abstract

We prove under commonly used assumptions the convergence of actor-critic reinforcement learning algorithms, which simultaneously learn a policy function, the actor, and a value function, the critic. Both functions can be deep neural networks of arbitrary complexity. Our framework allows showing convergence of the well known Proximal Policy Optimization (PPO) and of the recently introduced RUDDER. For the convergence proof we employ recently introduced techniques from the two time-scale stochastic approximation theory. Our results are valid for actor-critic methods that use episodic samples and that have a policy that becomes more greedy during learning. Previous convergence proofs assume linear function approximation, cannot treat episodic examples, or do not consider that policies become greedy. The latter is relevant since optimal policies are typically deterministic.