Efficient Counterfactual Learning from Bandit Feedback

TL;DR

This paper introduces statistically efficient estimators for off-policy evaluation in bandit settings, reducing variance and improving confidence in policy optimization from offline data, demonstrated through advertisement design improvements.

Contribution

The paper proposes new low-variance estimators for counterfactual evaluation that outperform standard methods in statistical efficiency and practical application.

Findings

Achieves lower variance than standard estimators.

Enables more confident policy improvements.

Demonstrates effectiveness in real-world advertisement optimization.

Abstract

What is the most statistically efficient way to do off-policy evaluation and optimization with batch data from bandit feedback? For log data generated by contextual bandit algorithms, we consider offline estimators for the expected reward from a counterfactual policy. Our estimators are shown to have lowest variance in a wide class of estimators, achieving variance reduction relative to standard estimators. We then apply our estimators to improve advertisement design by a major advertisement company. Consistent with the theoretical result, our estimators allow us to improve on the existing bandit algorithm with more statistical confidence compared to a state-of-the-art benchmark.