Policy Search: Any Local Optimum Enjoys a Global Performance Guarantee

TL;DR

This paper proves that any approximate local optimum in local policy search guarantees near-optimal global performance in reinforcement learning, challenging the common belief that only global optima can ensure good results.

Contribution

It establishes that all local optima in policy search have global performance guarantees, providing new theoretical insights into reinforcement learning optimization.

Findings

Any local optimum guarantees near-global performance.

Comparison shows local policy search can have better performance bounds than direct policy iteration.

The analysis impacts both practical algorithms and theoretical understanding of policy optimization.

Abstract

Local Policy Search is a popular reinforcement learning approach for handling large state spaces. Formally, it searches locally in a paramet erized policy space in order to maximize the associated value function averaged over some predefined distribution. It is probably commonly b elieved that the best one can hope in general from such an approach is to get a local optimum of this criterion. In this article, we show th e following surprising result: \emph{any} (approximate) \emph{local optimum} enjoys a \emph{global performance guarantee}. We compare this g uarantee with the one that is satisfied by Direct Policy Iteration, an approximate dynamic programming algorithm that does some form of Poli cy Search: if the approximation error of Local Policy Search may generally be bigger (because local search requires to consider a space of s tochastic policies), we argue that the concentrability coefficient that appears in the performance bound is much nicer. Finally, we discuss several practical and theoretical consequences of our analysis.