Rewarded Region Replay (R3) for Policy Learning with Discrete Action Space

TL;DR

The paper introduces Rewarded Region Replay (R3), an on-policy reinforcement learning algorithm that enhances sample efficiency and outperforms PPO and DDQN in discrete and dense reward environments by leveraging a replay buffer of successful trajectories.

Contribution

The paper proposes R3, a novel on-policy algorithm that uses a replay buffer of successful trajectories with importance sampling, reducing variance and improving performance in discrete and dense reward environments.

Findings

R3 outperforms PPO in Minigrid environments with sparse rewards.

R3 surpasses DDQN in DoorKeyEnv.

Dense R3 (DR3) outperforms PPO in Cartpole-V1.

Abstract

We introduce a new on-policy algorithm called Rewarded Region Replay (R3), which significantly improves on PPO in solving environments with discrete action spaces. R3 improves sample efficiency by using a replay buffer which contains past successful trajectories with reward above a certain threshold, which are used to update a PPO agent with importance sampling. Crucially, we discard the importance sampling factors which are above a certain ratio to reduce variance and stabilize training. We found that R3 significantly outperforms PPO in Minigrid environments with sparse rewards and discrete action space, such as DoorKeyEnv and CrossingEnv, and moreover we found that the improvement margin of our method versus baseline PPO increases with the complexity of the environment. We also benchmarked the performance of R3 against DDQN (Double Deep Q-Network), which is a standard baseline in off-policy methods for discrete actions, and found that R3 also outperforms DDQN agent in DoorKeyEnv. Lastly, we adapt the idea of R3 to dense reward setting to obtain the Dense R3 algorithm (or DR3) and benchmarked it against PPO on Cartpole-V1 environment. We found that DR3 outperforms PPO significantly on this dense reward environment. Our code can be found at https://github.com/chry-santhemum/R3.