Phasic Policy Gradient

TL;DR

Phasic Policy Gradient (PPG) is a reinforcement learning framework that enhances sample efficiency by separating policy and value function training into distinct phases, improving performance on complex benchmarks.

Contribution

PPG introduces a novel two-phase training approach that combines shared and separate network advantages for policy and value functions in reinforcement learning.

Findings

PPG significantly outperforms PPO in sample efficiency.

PPG achieves better results on the Procgen Benchmark.

The method effectively balances shared and separate training benefits.

Abstract

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.