Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio

TL;DR

This paper introduces PPO-RPE, a novel reinforcement learning method that uses a symmetric relative density ratio with an adaptive threshold to improve policy regularization and stability in complex environments.

Contribution

It proposes a new PPO variant utilizing relative Pearson divergence to adaptively set the threshold based on symmetry, enhancing policy update control.

Findings

The adaptive threshold improves policy regularization effectiveness.

PPO-RPE outperforms traditional methods in benchmark simulations.

The method statistically enhances task success in locomotion tasks.

Abstract

Deep reinforcement learning (DRL) is one of the promising approaches for introducing robots into complicated environments. The recent remarkable progress of DRL stands on regularization of policy, which allows the policy to improve stably and efficiently. A popular method, so-called proximal policy optimization (PPO), and its variants constrain density ratio of the latest and baseline policies when the density ratio exceeds a given threshold. This threshold can be designed relatively intuitively, and in fact its recommended value range has been suggested. However, the density ratio is asymmetric for its center, and the possible error scale from its center, which should be close to the threshold, would depend on how the baseline policy is given. In order to maximize the values of regularization of policy, this paper proposes a new PPO derived using relative Pearson (RPE) divergence, therefore so-called PPO-RPE, to design the threshold adaptively. In PPO-RPE, the relative density ratio, which can be formed with symmetry, replaces the raw density ratio. Thanks to this symmetry, its error scale from center can easily be estimated, hence, the threshold can be adapted for the estimated error scale. From three simple benchmark simulations, the importance of algorithm-dependent threshold design is revealed. By simulating additional four locomotion tasks, it is verified that the proposed method statistically contributes to task accomplishment by appropriately restricting the policy updates.