Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters

TL;DR

This paper investigates the importance of calibration in estimating behaviour policies for off-policy evaluation, demonstrating that simple non-parametric models can outperform neural networks in calibration and OPE accuracy.

Contribution

It highlights the critical role of calibration in behaviour policy estimation and shows that non-parametric models can yield better calibrated estimates for OPE.

Findings

Neural networks can produce highly uncalibrated behaviour policy models.

Non-parametric k-nearest neighbors models achieve better calibration.

Better calibration leads to more accurate importance sampling-based OPE.

Abstract

In this work, we consider the problem of estimating a behaviour policy for use in Off-Policy Policy Evaluation (OPE) when the true behaviour policy is unknown. Via a series of empirical studies, we demonstrate how accurate OPE is strongly dependent on the calibration of estimated behaviour policy models: how precisely the behaviour policy is estimated from data. We show how powerful parametric models such as neural networks can result in highly uncalibrated behaviour policy models on a real-world medical dataset, and illustrate how a simple, non-parametric, k-nearest neighbours model produces better calibrated behaviour policy estimates and can be used to obtain superior importance sampling-based OPE estimates.