SOPE: Spectrum of Off-Policy Estimators

TL;DR

This paper introduces a spectrum of off-policy estimators that balance bias and variance, improving the accuracy of policy evaluation in sequential decision-making tasks.

Contribution

It presents a unified framework connecting importance sampling and state-action distribution methods, enabling better bias-variance trade-offs.

Findings

Estimators in the spectrum can outperform traditional IS and SIS in mean-squared error.

The spectrum includes endpoints corresponding to IS and SIS, as well as intermediate estimators.

Empirical results demonstrate improved policy evaluation accuracy.

Abstract

Many sequential decision making problems are high-stakes and require off-policy evaluation (OPE) of a new policy using historical data collected using some other policy. One of the most common OPE techniques that provides unbiased estimates is trajectory based importance sampling (IS). However, due to the high variance of trajectory IS estimates, importance sampling methods based on state-action visitation distributions (SIS) have recently been adopted. Unfortunately, while SIS often provides lower variance estimates for long horizons, estimating the state-action distribution ratios can be challenging and lead to biased estimates. In this paper, we present a new perspective on this bias-variance trade-off and show the existence of a spectrum of estimators whose endpoints are SIS and IS. Additionally, we also establish a spectrum for doubly-robust and weighted version of these estimators. We provide empirical evidence that estimators in this spectrum can be used to trade-off between the bias and variance of IS and SIS and can achieve lower mean-squared error than both IS and SIS.