Policy Optimization via Importance Sampling

TL;DR

This paper introduces POIS, a new model-free policy search algorithm for reinforcement learning that uses importance sampling and high-confidence bounds to improve trajectory reuse and optimize policies efficiently in continuous control tasks.

Contribution

The paper presents POIS, a novel policy optimization algorithm that adaptively balances online and offline updates using importance sampling bounds, applicable to both action and parameter-based policies.

Findings

POIS outperforms state-of-the-art methods on continuous control benchmarks.

The high-confidence importance sampling bound effectively guides policy updates.

POIS is versatile, working with linear and deep policies.

Abstract

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.