An operator view of policy gradient methods

TL;DR

This paper introduces an operator-based framework for policy gradient methods, providing new insights into their mechanisms, proposing a global lower bound of expected return, and bridging the gap between policy-based and value-based reinforcement learning approaches.

Contribution

It presents a novel operator perspective on policy gradient methods, leading to a better understanding and a new global lower bound for expected return.

Findings

Operator framework clarifies policy gradient methods

Introduces a new global lower bound of expected return

Bridges policy-based and value-based methods

Abstract

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator $\mathcal{I}$, which maps any policy $\pi$ to a better one $\mathcal{I}\pi$, and a projection operator $\mathcal{P}$, which finds the best approximation of $\mathcal{I}\pi$ in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of $\mathcal{I}$ and $\mathcal{P}$ to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.