Adversarial Bandits against Arbitrary Strategies

TL;DR

This paper addresses adversarial bandit problems with arbitrary strategies by developing algorithms that adapt to the number of switches in the best arm, achieving improved regret bounds through adaptive methods.

Contribution

The paper introduces a master-base framework with adaptive learning rates for online mirror descent, leading to tighter regret bounds in adversarial bandit settings with switches.

Findings

Achieved regret of O(S^{1/2}K^{1/3}T^{2/3}) with simple OMD.

Improved regret to O(\u221a{ ext{min}\{ ext{SKT} ho, ext{S} ext{KT} ight}) using adaptive learning rates.

Demonstrated effectiveness of adaptive methods in handling variance in loss estimators.

Abstract

We study the adversarial bandit problem against arbitrary strategies, where the difficulty is captured by an unknown parameter $S$, which is the number of switches in the best arm in hindsight. To handle this problem, we adopt the master-base framework using the online mirror descent method (OMD). We first provide a master-base algorithm with simple OMD, achieving $\tilde{O}(S^{1/2}K^{1/3}T^{2/3})$, in which $T^{2/3}$ comes from the variance of loss estimators. To mitigate the impact of the variance, we propose using adaptive learning rates for OMD and achieve $\tilde{O}(\min\{\sqrt{SKT\rho},S\sqrt{KT}\})$, where $\rho$ is a variance term for loss estimators.