Efficient iterative policy optimization

TL;DR

This paper introduces an efficient iterative policy optimization method that reduces the number of updates needed by approximating expected rewards with concave bounds, and extends techniques to handle negative rewards with control variates.

Contribution

It presents a novel approach to policy optimization that minimizes updates through concave lower bounds and incorporates negative rewards using control variates.

Findings

Significantly fewer policy updates needed for good performance

Effective handling of negative rewards in policy optimization

Enhanced optimization efficiency compared to existing methods

Abstract

We tackle the issue of finding a good policy when the number of policy updates is limited. This is done by approximating the expected policy reward as a sequence of concave lower bounds which can be efficiently maximized, drastically reducing the number of policy updates required to achieve good performance. We also extend existing methods to negative rewards, enabling the use of control variates.