The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond

TL;DR

This paper introduces the KL-UCB algorithm for stochastic bandits, demonstrating its superior regret bounds and efficiency over existing methods, especially for bounded and Bernoulli rewards, supported by theoretical analysis and numerical experiments.

Contribution

The paper provides a finite-time analysis of KL-UCB, proving its optimality for Bernoulli rewards and extending its applicability to various reward distributions with improved regret bounds.

Findings

KL-UCB outperforms UCB and UCB2 in regret bounds.

KL-UCB achieves the Lai and Robbins lower bound for Bernoulli rewards.

Numerical results show KL-UCB's efficiency and stability across different scenarios.

Abstract

This paper presents a finite-time analysis of the KL-UCB algorithm, an online, horizon-free index policy for stochastic bandit problems. We prove two distinct results: first, for arbitrary bounded rewards, the KL-UCB algorithm satisfies a uniformly better regret bound than UCB or UCB2; second, in the special case of Bernoulli rewards, it reaches the lower bound of Lai and Robbins. Furthermore, we show that simple adaptations of the KL-UCB algorithm are also optimal for specific classes of (possibly unbounded) rewards, including those generated from exponential families of distributions. A large-scale numerical study comparing KL-UCB with its main competitors (UCB, UCB2, UCB-Tuned, UCB-V, DMED) shows that KL-UCB is remarkably efficient and stable, including for short time horizons. KL-UCB is also the only method that always performs better than the basic UCB policy. Our regret bounds rely on deviations results of independent interest which are stated and proved in the Appendix. As a by-product, we also obtain an improved regret bound for the standard UCB algorithm.