Double Clipping: Less-Biased Variance Reduction in Off-Policy Evaluation

TL;DR

This paper introduces double clipping, a simple extension to importance weight clipping in off-policy evaluation, which reduces bias while maintaining variance reduction, leading to more accurate estimations of expected rewards.

Contribution

The paper proposes double clipping, a novel technique that compensates for the downward bias of traditional clipping, improving the accuracy of off-policy estimators.

Findings

Double clipping reduces bias compared to standard clipping.

The method maintains variance reduction benefits.

Experimental results demonstrate improved estimation accuracy.

Abstract

"Clipping" (a.k.a. importance weight truncation) is a widely used variance-reduction technique for counterfactual off-policy estimators. Like other variance-reduction techniques, clipping reduces variance at the cost of increased bias. However, unlike other techniques, the bias introduced by clipping is always a downward bias (assuming non-negative rewards), yielding a lower bound on the true expected reward. In this work we propose a simple extension, called $\textit{double clipping}$, which aims to compensate this downward bias and thus reduce the overall bias, while maintaining the variance reduction properties of the original estimator.