Algorithms for Linear Bandits on Polyhedral Sets

TL;DR

This paper introduces nearly optimal algorithms for stochastic linear bandit problems within polyhedral sets, achieving logarithmic regret bounds and addressing open questions about efficient exploration strategies.

Contribution

It provides a nearly optimal algorithm with logarithmic regret bounds for linear bandits on polyhedral sets, resolving an open problem and demonstrating robustness to reward perturbations.

Findings

Expected regret scales as (N    T)

Proposed algorithms achieve regret of O(N       (T))

Algorithms perform well in finite time, matching theoretical bounds asymptotically.

Abstract

We study stochastic linear optimization problem with bandit feedback. The set of arms take values in an $N$-dimensional space and belong to a bounded polyhedron described by finitely many linear inequalities. We provide a lower bound for the expected regret that scales as $\Omega(N\log T)$. We then provide a nearly optimal algorithm and show that its expected regret scales as $O(N\log^{1+\epsilon}(T))$ for an arbitrary small $\epsilon >0$. The algorithm alternates between exploration and exploitation intervals sequentially where deterministic set of arms are played in the exploration intervals and greedily selected arm is played in the exploitation intervals. We also develop an algorithm that achieves the optimal regret when sub-Gaussianity parameter of the noise term is known. Our key insight is that for a polyhedron the optimal arm is robust to small perturbations in the reward function. Consequently, a greedily selected arm is guaranteed to be optimal when the estimation error falls below some suitable threshold. Our solution resolves a question posed by Rusmevichientong and Tsitsiklis (2011) that left open the possibility of efficient algorithms with asymptotic logarithmic regret bounds. We also show that the regret upper bounds hold with probability $1$. Our numerical investigations show that while theoretical results are asymptotic the performance of our algorithms compares favorably to state-of-the-art algorithms in finite time as well.