Efficient Multi-Policy Evaluation for Reinforcement Learning

TL;DR

This paper introduces a novel approach for efficiently evaluating multiple policies in reinforcement learning by designing a tailored behavior policy that reduces variance and improves sample efficiency, outperforming existing methods.

Contribution

The paper proposes a new behavior policy for multi-policy evaluation that reduces variance and sample complexity, with theoretical guarantees and empirical validation.

Findings

Lower variance compared to previous methods

Fewer samples needed for accurate evaluation

State-of-the-art performance across various environments

Abstract

To unbiasedly evaluate multiple target policies, the dominant approach among RL practitioners is to run and evaluate each target policy separately. However, this evaluation method is far from efficient because samples are not shared across policies, and running target policies to evaluate themselves is actually not optimal. In this paper, we address these two weaknesses by designing a tailored behavior policy to reduce the variance of estimators across all target policies. Theoretically, we prove that executing this behavior policy with manyfold fewer samples outperforms on-policy evaluation on every target policy under characterized conditions. Empirically, we show our estimator has a substantially lower variance compared with previous best methods and achieves state-of-the-art performance in a broad range of environments.