An Approximate Ascent Approach To Prove Convergence of PPO

TL;DR

This paper provides a theoretical convergence proof for PPO by interpreting its policy updates as approximate policy gradient ascent and addresses issues in advantage estimation, supported by empirical improvements.

Contribution

It introduces a novel convergence analysis of PPO using approximate ascent theory and identifies a bias issue in advantage estimation, proposing a correction method.

Findings

Convergence of PPO can be proven under standard assumptions.

A bias in advantage estimation can cause issues at episode boundaries.

Correcting advantage weights improves performance in terminal environments.

Abstract

Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling to prove a convergence theorem for PPO that sheds light on PPO's success. Additionally, we identify a previously overlooked issue in truncated Generalized Advantage Estimation commonly used in PPO. The geometric weighting scheme induces infinite mass collapse onto the longest $k$-step advantage estimator at episode boundaries. Empirical evaluations show that a simple weight correction can yield substantial improvements in environments with strong terminal signal, such as Lunar Lander.