A Simple and Optimal Policy Design with Safety against Heavy-Tailed Risk for Stochastic Bandits

TL;DR

This paper introduces a new policy for stochastic multi-armed bandits that achieves optimal worst-case expected regret and provides strong tail bounds on regret distribution, balancing exploration and exploitation effectively.

Contribution

The paper proposes a novel policy that is both worst-case optimal for expected regret and has the best possible tail probability bounds, with extensions to unknown horizon and linear bandits.

Findings

Achieves $O(\sqrt{KT\ln T})$ expected regret bound.

Provides exponential tail bounds on regret distribution.

Outperforms existing policies in tail risk and hyper-parameter tuning.

Abstract

We study the stochastic multi-armed bandit problem and design new policies that enjoy both worst-case optimality for expected regret and light-tailed risk for regret distribution. Specifically, our policy design (i) enjoys the worst-case optimality for the expected regret at order $O(\sqrt{KT\ln T})$ and (ii) has the worst-case tail probability of incurring a regret larger than any $x>0$ being upper bounded by $\exp(-\Omega(x/\sqrt{KT}))$, a rate that we prove to be best achievable with respect to $T$ for all worst-case optimal policies. Our proposed policy achieves a delicate balance between doing more exploration at the beginning of the time horizon and doing more exploitation when approaching the end, compared to standard confidence-bound-based policies. We also enhance the policy design to accommodate the "any-time" setting where $T$ is unknown a priori, and prove equivalently desired policy performances as compared to the "fixed-time" setting with known $T$. Numerical experiments are conducted to illustrate the theoretical findings. We find that from a managerial perspective, our new policy design yields better tail distributions and is preferable than celebrated policies especially when (i) there is a risk of under-estimating the volatility profile, or (ii) there is a challenge of tuning policy hyper-parameters. We conclude by extending our proposed policy design to the stochastic linear bandit setting that leads to both worst-case optimality in terms of expected regret and light-tailed risk on the regret distribution.