Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits

TL;DR

This paper introduces the Marginal Ratio (MR) estimator for off-policy evaluation in contextual bandits, reducing variance compared to existing methods by focusing on outcome distribution shifts, with proven theoretical and practical benefits.

Contribution

The paper proposes the MR estimator, a novel approach that improves variance reduction in OPE by shifting focus to outcome distribution changes, and establishes its superiority over existing estimators like IPW, DR, and MIPS.

Findings

MR estimator achieves lower variance than IPW and DR.

Theoretical analysis confirms MR's variance reduction benefits.

Experiments demonstrate MR's practical effectiveness in real-world datasets.

Abstract

Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation. However, current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators, suffer from high variance, particularly in cases of low overlap between target and behavior policies or large action and context spaces. In this paper, we introduce a new OPE estimator for contextual bandits, the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal distribution of outcomes $Y$ instead of the policies themselves. Through rigorous theoretical analysis, we demonstrate the benefits of the MR estimator compared to conventional methods like IPW and DR in terms of variance reduction. Additionally, we establish a connection between the MR estimator and the state-of-the-art Marginalized Inverse Propensity Score (MIPS) estimator, proving that MR achieves lower variance among a generalized family of MIPS estimators. We further illustrate the utility of the MR estimator in causal inference settings, where it exhibits enhanced performance in estimating Average Treatment Effects (ATE). Our experiments on synthetic and real-world datasets corroborate our theoretical findings and highlight the practical advantages of the MR estimator in OPE for contextual bandits.