Empirical Likelihood for Contextual Bandits

TL;DR

This paper introduces a new empirical likelihood-based estimator and confidence interval for off-policy evaluation in contextual bandits, enabling more reliable policy value estimation and optimization from limited data.

Contribution

It presents a novel empirical likelihood approach for off-policy evaluation and a policy optimization method based on the lower confidence bound, improving over previous methods.

Findings

Estimator and confidence interval outperform previous methods in finite samples

The policy optimization algorithm finds policies with higher reward lower bounds

Empirical results demonstrate improved off-policy learning performance

Abstract

We propose an estimator and confidence interval for computing the value of a policy from off-policy data in the contextual bandit setting. To this end we apply empirical likelihood techniques to formulate our estimator and confidence interval as simple convex optimization problems. Using the lower bound of our confidence interval, we then propose an off-policy policy optimization algorithm that searches for policies with large reward lower bound. We empirically find that both our estimator and confidence interval improve over previous proposals in finite sample regimes. Finally, the policy optimization algorithm we propose outperforms a strong baseline system for learning from off-policy data.