On Policy Gradients

TL;DR

This paper provides a formal introduction to policy gradient methods, explaining their development, challenges with sample efficiency, and their role in optimizing policies in reinforcement learning.

Contribution

It offers a comprehensive overview of policy gradient approaches, clarifying their theoretical foundations and highlighting the importance of improving sample efficiency.

Findings

Policy gradients are used to optimize policies by gradient ascent.

Sample efficiency is a major challenge in policy gradient methods.

The paper clarifies the development and theoretical basis of policy gradient algorithms.

Abstract

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The main issue is that the policy gradient with respect to the expected return is not available, thus we need to estimate it. As policy gradient algorithms also tend to require on-policy data for the gradient estimate, their biggest weakness is sample efficiency. For this reason, most research is focused on finding algorithms with improved sample efficiency. This paper provides a formal introduction to policy gradient that shows the development of policy gradient approaches, and should enable the reader to follow current research on the topic.