Contextual Bandit Algorithms with Supervised Learning Guarantees

TL;DR

This paper introduces new algorithms for contextual bandit problems that achieve strong theoretical guarantees and perform well empirically, bridging the gap between bandit and supervised learning.

Contribution

It presents two novel algorithms, Exp4.P and VE, with improved regret bounds and guarantees in both finite and infinite policy classes, advancing theoretical understanding and practical performance.

Findings

Exp4.P competes with the best expert with high probability and low regret.

VE algorithm handles infinite policy classes with VC-dimension bounds.

Empirical tests on large-scale datasets validate the algorithms' effectiveness.

Abstract

We address the problem of learning in an online, bandit setting where the learner must repeatedly select among $K$ actions, but only receives partial feedback based on its choices. We establish two new facts: First, using a new algorithm called Exp4.P, we show that it is possible to compete with the best in a set of $N$ experts with probability $1-\delta$ while incurring regret at most $O(\sqrt{KT\ln(N/\delta)})$ over $T$ time steps. The new algorithm is tested empirically in a large-scale, real-world dataset. Second, we give a new algorithm called VE that competes with a possibly infinite set of policies of VC-dimension $d$ while incurring regret at most $O(\sqrt{T(d\ln(T) + \ln (1/\delta))})$ with probability $1-\delta$. These guarantees improve on those of all previous algorithms, whether in a stochastic or adversarial environment, and bring us closer to providing supervised learning type guarantees for the contextual bandit setting.