An Improved Algorithm for Adversarial Linear Contextual Bandits via Reduction

TL;DR

This paper introduces an efficient algorithm for adversarial linear contextual bandits that achieves near-optimal regret bounds without prior knowledge of context distributions, resolving key open questions in the field.

Contribution

The authors develop a reduction-based algorithm that attains poly(d)√T regret in polynomial time, even with adversarial losses and stochastic action sets, improving upon previous methods.

Findings

Achieves  O( ext{min}\{d^2 ext{ }\sqrt{T}, ext{ }\sqrt{d^3T ext{ }\log K} ight\u007F O) regret.

First polynomial-time algorithm for combinatorial bandits with adversarial losses and stochastic action sets.

Improves regret bounds to  O(d ext{ }\sqrt{L^\u00D7}) with a simulator.

Abstract

We present an efficient algorithm for linear contextual bandits with adversarial losses and stochastic action sets. Our approach reduces this setting to misspecification-robust adversarial linear bandits with fixed action sets. Without knowledge of the context distribution or access to a context simulator, the algorithm achieves $\tilde{O}(\min\{d^2\sqrt{T}, \sqrt{d^3T\log K}\})$ regret and runs in $\text{poly}(d,C,T)$ time, where $d$ is the feature dimension, $C$ is an upper bound on the number of linear constraints defining the action set in each round, $K$ is an upper bound on the number of actions in each round, and $T$ is number of rounds. This resolves the open question by Liu et al. (2023) on whether one can obtain $\text{poly}(d)\sqrt{T}$ regret in polynomial time independent of the number of actions. For the important class of combinatorial bandits with adversarial losses and stochastic action sets where the action sets can be described by a polynomial number of linear constraints, our algorithm is the first to achieve $\text{poly}(d)\sqrt{T}$ regret in polynomial time, while no prior algorithm achieves even $o(T)$ regret in polynomial time to our knowledge. When a simulator is available, the regret bound can be improved to $\tilde{O}(d\sqrt{L^\star})$, where $L^\star$ is the cumulative loss of the best policy.