Differentiable Linear Bandit Algorithm

TL;DR

This paper introduces a differentiable linear bandit algorithm that learns the confidence bound adaptively from data, improving exploration-exploitation balance and outperforming baselines in experiments.

Contribution

It proposes the first differentiable linear bandit algorithm with a gradient-based method to learn confidence bounds directly from data.

Findings

The learned confidence bound $\hat{eta}$ is significantly smaller than theoretical bounds.

The proposed method achieves near-optimal regret bounds.

Empirical results outperform baseline algorithms on multiple datasets.

Abstract

Upper Confidence Bound (UCB) is arguably the most commonly used method for linear multi-arm bandit problems. While conceptually and computationally simple, this method highly relies on the confidence bounds, failing to strike the optimal exploration-exploitation if these bounds are not properly set. In the literature, confidence bounds are typically derived from concentration inequalities based on assumptions on the reward distribution, e.g., sub-Gaussianity. The validity of these assumptions however is unknown in practice. In this work, we aim at learning the confidence bound in a data-driven fashion, making it adaptive to the actual problem structure. Specifically, noting that existing UCB-typed algorithms are not differentiable with respect to confidence bound, we first propose a novel differentiable linear bandit algorithm. Then, we introduce a gradient estimator, which allows the confidence bound to be learned via gradient ascent. Theoretically, we show that the proposed algorithm achieves a $\tilde{\mathcal{O}}(\hat{\beta}\sqrt{dT})$ upper bound of $T$-round regret, where $d$ is the dimension of arm features and $\hat{\beta}$ is the learned size of confidence bound. Empirical results show that $\hat{\beta}$ is significantly smaller than its theoretical upper bound and proposed algorithms outperforms baseline ones on both simulated and real-world datasets.