$\Delta\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies

TL;DR

This paper introduces $ ext{-}{ m OPE}$, a pairwise off-policy estimation method that reduces variance in policy value difference estimation, improving offline evaluation and learning in recommendation systems.

Contribution

The paper proposes $ ext{-}{ m OPE}$, a novel pairwise off-policy estimation framework that leverages covariance between policies to reduce variance and enhance efficiency.

Findings

$ ext{-}{ m OPE}$ improves estimation accuracy in simulations and real experiments.

Variance reduction leads to better policy evaluation and learning outcomes.

The method outperforms traditional estimators in offline and online settings.

Abstract

The off-policy paradigm casts recommendation as a counterfactual decision-making task, allowing practitioners to unbiasedly estimate online metrics using offline data. This leads to effective evaluation metrics, as well as learning procedures that directly optimise online success. Nevertheless, the high variance that comes with unbiasedness is typically the crux that complicates practical applications. An important insight is that the difference between policy values can often be estimated with significantly reduced variance, if said policies have positive covariance. This allows us to formulate a pairwise off-policy estimation task: $\Delta\text{-}{\rm OPE}$. $\Delta\text{-}{\rm OPE}$ subsumes the common use-case of estimating improvements of a learnt policy over a production policy, using data collected by a stochastic logging policy. We introduce $\Delta\text{-}{\rm OPE}$ methods based on the widely used Inverse Propensity Scoring estimator and its extensions. Moreover, we characterise a variance-optimal additive control variate that further enhances efficiency. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.