Optimistic Policy Regularization

TL;DR

This paper introduces Optimistic Policy Regularization (OPR), a method that enhances deep reinforcement learning by maintaining successful trajectories, leading to improved sample efficiency and performance across Atari games and cyber-defense environments.

Contribution

The paper proposes OPR, a novel regularization technique that preserves successful behaviors during policy training, significantly boosting sample efficiency and performance in deep reinforcement learning.

Findings

OPR improves sample efficiency on Atari games.

OPR achieves higher scores in 22 out of 49 Atari environments.

OPR outperforms baseline methods in cyber-defense tasks.

Abstract

Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.