Beyond the Boundaries of Proximal Policy Optimization

TL;DR

This paper reinterprets PPO by decomposing it into separate estimation and application steps, introduces outer-PPO allowing flexible optimizers, and empirically demonstrates improved performance with non-unity learning rates and momentum.

Contribution

It proposes outer-PPO, a new framework that decouples update estimation from application, enabling the use of arbitrary optimizers and challenging PPO's implicit design choices.

Findings

Non-unity learning rates improve performance on Brax and Jumanji environments.

Momentum applied to the outer loop yields statistically significant gains.

Decoupling update estimation and application reveals new optimization strategies for PPO.

Abstract

Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget.