Communication-Constrained Bandits under Additive Gaussian Noise

TL;DR

This paper investigates the fundamental limits and proposes an optimal algorithm for distributed multi-armed bandits with communication constraints and Gaussian noise, achieving near-minimax regret bounds.

Contribution

It derives a tight information-theoretic lower bound and introduces the $ exttt{UE-}UCB++$ algorithm that nearly attains this bound in a communication-constrained noisy setting.

Findings

Lower bound on regret: $oxed{ ilde{ ext{O}}ig(rac{ ext{poly}(K)}{ ext{SNR}}ig)}$

Proposed $ exttt{UE-}UCB++$ matches the lower bound up to a small additive factor

Algorithm effectively refines reward estimates through phased exploration and encoding.

Abstract

We study a distributed stochastic multi-armed bandit where a client supplies the learner with communication-constrained feedback based on the rewards for the corresponding arm pulls. In our setup, the client must encode the rewards such that the second moment of the encoded rewards is no more than $P$, and this encoded reward is further corrupted by additive Gaussian noise of variance $\sigma^2$; the learner only has access to this corrupted reward. For this setting, we derive an information-theoretic lower bound of $\Omega\left(\sqrt{\frac{KT}{\mathtt{SNR} \wedge1}} \right)$ on the minimax regret of any scheme, where $ \mathtt{SNR} := \frac{P}{\sigma^2}$, and $K$ and $T$ are the number of arms and time horizon, respectively. Furthermore, we propose a multi-phase bandit algorithm, $\mathtt{UE\text{-}UCB++}$, which matches this lower bound to a minor additive factor. $\mathtt{UE\text{-}UCB++}$ performs uniform exploration in its initial phases and then utilizes the {\em upper confidence bound }(UCB) bandit algorithm in its final phase. An interesting feature of $\mathtt{UE\text{-}UCB++}$ is that the coarser estimates of the mean rewards formed during a uniform exploration phase help to refine the encoding protocol in the next phase, leading to more accurate mean estimates of the rewards in the subsequent phase. This positive reinforcement cycle is critical to reducing the number of uniform exploration rounds and closely matching our lower bound.