PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay

TL;DR

PTR-PPO enhances on-policy reinforcement learning by integrating prioritized trajectory replay, combining on-policy and off-policy methods to improve data efficiency and achieve state-of-the-art results in Atari tasks.

Contribution

This paper introduces PTR-PPO, a novel algorithm that incorporates prioritized trajectory replay into PPO, with new priority measures and variance reduction techniques.

Findings

Achieves state-of-the-art performance on Atari tasks.

Memory size and rollout length significantly affect priority distribution.

Prioritized replay improves sampling efficiency in reinforcement learning.

Abstract

On-policy deep reinforcement learning algorithms have low data utilization and require significant experience for policy improvement. This paper proposes a proximal policy optimization algorithm with prioritized trajectory replay (PTR-PPO) that combines on-policy and off-policy methods to improve sampling efficiency by prioritizing the replay of trajectories generated by old policies. We first design three trajectory priorities based on the characteristics of trajectories: the first two being max and mean trajectory priorities based on one-step empirical generalized advantage estimation (GAE) values and the last being reward trajectory priorities based on normalized undiscounted cumulative reward. Then, we incorporate the prioritized trajectory replay into the PPO algorithm, propose a truncated importance weight method to overcome the high variance caused by large importance weights under multistep experience, and design a policy improvement loss function for PPO under off-policy conditions. We evaluate the performance of PTR-PPO in a set of Atari discrete control tasks, achieving state-of-the-art performance. In addition, by analyzing the heatmap of priority changes at various locations in the priority memory during training, we find that memory size and rollout length can have a significant impact on the distribution of trajectory priorities and, hence, on the performance of the algorithm.