Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezing

TL;DR

This paper introduces a new off-policy selection method using betting-based confidence bounds and a generic condition called freezing, leading to improved variance-adaptive guarantees and better performance in small-data scenarios.

Contribution

It proposes a novel betting-based confidence bound for off-policy selection and a generic condition called freezing that balances bias and variance.

Findings

The new selection method outperforms existing methods.

Freezing achieves low variance in small-sample regimes.

Theoretical guarantees are improved and variance-adaptive.

Abstract

We consider off-policy selection and learning in contextual bandits, where the learner aims to select or train a reward-maximizing policy using data collected by a fixed behavior policy. Our contribution is two-fold. First, we propose a novel off-policy selection method that leverages a new betting-based confidence bound applied to an inverse propensity weight sequence. Our theoretical analysis reveals that this method achieves a significantly improved, variance-adaptive guarantee over prior work. Second, we propose a novel and generic condition on the optimization objective for off-policy learning that strikes a different balance between bias and variance. One special case, which we call freezing, tends to induce low variance, which is preferred in small-data regimes. Our analysis shows that it matches the best existing guarantees. In our empirical study, our selection method outperforms existing methods, and freezing exhibits improved performance in small-sample regimes.