Trust Region Policy Optimization

TL;DR

Trust Region Policy Optimization (TRPO) is a practical reinforcement learning algorithm that guarantees monotonic policy improvement and performs well across diverse tasks like robotics and Atari games.

Contribution

The paper introduces TRPO, a new policy optimization method that balances theoretical guarantees with practical effectiveness for large nonlinear policies.

Findings

TRPO achieves robust performance on robotic control tasks.

TRPO effectively learns policies from high-dimensional image inputs.

The algorithm maintains monotonic improvement despite approximations.

Abstract

We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.