Maximum Entropy Hindsight Experience Replay

TL;DR

This paper enhances goal-based reinforcement learning by improving the application of Hindsight Experience Replay (HER) in on-policy algorithms like PPO, leading to more efficient learning in Predator-Prey environments.

Contribution

The paper introduces a principled method for selectively applying HER to on-policy algorithms, improving upon previous PPO-HER implementations.

Findings

Selective HER application accelerates learning.

Improved PPO-HER outperforms previous methods.

Enhanced goal-based RL efficiency.

Abstract

Hindsight experience replay (HER) is well-known to accelerate goal-based reinforcement learning (RL). While HER is generally applied to off-policy RL algorithms, we previously showed that HER can also accelerate on-policy algorithms, such as proximal policy optimization (PPO), for goal-based Predator-Prey environments. Here, we show that we can improve the previous PPO-HER algorithm by selectively applying HER in a principled manner.