Learning to Constrain Policy Optimization with Virtual Trust Region

TL;DR

This paper presents MCPO, a new reinforcement learning method that uses virtual trust regions built from past policies to improve policy updates, especially when previous policies perform poorly, showing strong results across various tasks.

Contribution

Introducing a dynamic virtual trust region mechanism in policy optimization that leverages past policies for improved reinforcement learning performance.

Findings

MCPO outperforms recent on-policy constrained methods in diverse environments.

The virtual trust region mechanism adapts effectively during training.

Memory-based virtual policies enhance policy stability and learning efficiency.

Abstract

We introduce a constrained optimization method for policy gradient reinforcement learning, which uses a virtual trust region to regulate each policy update. In addition to using the proximity of one single old policy as the normal trust region, we propose forming a second trust region through another virtual policy representing a wide range of past policies. We then enforce the new policy to stay closer to the virtual policy, which is beneficial if the old policy performs poorly. More importantly, we propose a mechanism to automatically build the virtual policy from a memory of past policies, providing a new capability for dynamically learning appropriate virtual trust regions during the optimization process. Our proposed method, dubbed Memory-Constrained Policy Optimization (MCPO), is examined in diverse environments, including robotic locomotion control, navigation with sparse rewards and Atari games, consistently demonstrating competitive performance against recent on-policy constrained policy gradient methods.