CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

TL;DR

This paper introduces a family of off-policy evaluation estimators that incorporate counterfactual annotations using a doubly robust approach, improving accuracy and robustness in realistic, imperfect annotation scenarios.

Contribution

It proposes three strategies for integrating counterfactual annotations into doubly robust estimators and provides theoretical and empirical guidance on their optimal use.

Findings

Using imperfect annotations in the reward model (DM) improves estimator performance.

The proposed methods outperform traditional importance sampling in scenarios with model misspecification.

Empirical results validate the theoretical advantage of using annotations in the DM component.

Abstract

Off-policy evaluation (OPE) provides safety guarantees by estimating the performance of a policy before deployment. Recent work introduced IS+, an importance sampling (IS) estimator that uses expert-annotated counterfactual samples to improve behavior dataset coverage. However, IS estimators are known to have high variance; furthermore, the performance of IS+ deteriorates when annotations are imperfect. In this work, we propose a family of OPE estimators inspired by the doubly robust (DR) principle. A DR estimator combines IS with a reward model estimate, known as the direct method (DM), and offers favorable statistical guarantees. We propose three strategies for incorporating counterfactual annotations into a DR-inspired estimator and analyze their properties under various realistic settings. We prove that using imperfect annotations in the DM part of the estimator best leverages the annotations, as opposed to using them in the IS part. To support our theoretical findings, we evaluate the proposed estimators in three contextual bandit environments. Our empirical results show that when the reward model is misspecified and the annotations are imperfect, it is most beneficial to use the annotations only in the DM portion of a DR estimator. Based on these theoretical and empirical insights, we provide a practical guide for using counterfactual annotations in different realistic settings.