Multi-agent Multi-armed Bandit with Fully Heavy-tailed Dynamics

TL;DR

This paper introduces algorithms for decentralized multi-agent multi-armed bandits in fully heavy-tailed environments, addressing challenges in communication and inference with heavy-tailed rewards and sparse graph structures, achieving near-optimal regret bounds.

Contribution

First to analyze multi-agent bandits with fully heavy-tailed reward distributions and sparse graph communication, providing new algorithms with provable regret bounds.

Findings

Achieved regret bounds of $O(M^{1 - 1/\alpha} \log T)$ for homogeneous rewards.

Established $O(M \log T)$ regret bounds for heterogeneous reward settings.

Utilized hub-like structures and new information delay bounds to improve performance.

Abstract

We study decentralized multi-agent multi-armed bandits in fully heavy-tailed settings, where clients communicate over sparse random graphs with heavy-tailed degree distributions and observe heavy-tailed (homogeneous or heterogeneous) reward distributions with potentially infinite variance. The objective is to maximize system performance by pulling the globally optimal arm with the highest global reward mean across all clients. We are the first to address such fully heavy-tailed scenarios, which capture the dynamics and challenges in communication and inference among multiple clients in real-world systems. In homogeneous settings, our algorithmic framework exploits hub-like structures unique to heavy-tailed graphs, allowing clients to aggregate rewards and reduce noises via hub estimators when constructing UCB indices; under $M$ clients and degree distributions with power-law index $\alpha > 1$, our algorithm attains a regret bound (almost) of order $O(M^{1 -\frac{1}{\alpha}} \log{T})$. Under heterogeneous rewards, clients synchronize by communicating with neighbors, aggregating exchanged estimators in UCB indices; With our newly established information delay bounds on sparse random graphs, we prove a regret bound of $O(M \log{T})$. Our results improve upon existing work, which only address time-invariant connected graphs, or light-tailed dynamics in dense graphs and rewards.