Decentralized Randomly Distributed Multi-agent Multi-armed Bandit with Heterogeneous Rewards

TL;DR

This paper introduces a decentralized multi-agent multi-armed bandit algorithm that effectively manages heterogeneous rewards and random communication graphs, achieving near-optimal regret bounds with high probability.

Contribution

It proposes a novel framework combining robust graph simulation, consensus averaging, and UCB techniques, removing the need for doubly stochastic graphs and handling diverse reward distributions.

Findings

Achieves optimal $ ext{log} T$ regret bounds in sub-Gaussian and sub-exponential environments.

Provides high-probability regret bounds that account for graph randomness.

Handles heterogeneous rewards across clients without requiring prior knowledge of the reward distributions.

Abstract

We study a decentralized multi-agent multi-armed bandit problem in which multiple clients are connected by time dependent random graphs provided by an environment. The reward distributions of each arm vary across clients and rewards are generated independently over time by an environment based on distributions that include both sub-exponential and sub-gaussian distributions. Each client pulls an arm and communicates with neighbors based on the graph provided by the environment. The goal is to minimize the overall regret of the entire system through collaborations. To this end, we introduce a novel algorithmic framework, which first provides robust simulation methods for generating random graphs using rapidly mixing Markov chains or the random graph model, and then combines an averaging-based consensus approach with a newly proposed weighting technique and the upper confidence bound to deliver a UCB-type solution. Our algorithms account for the randomness in the graphs, removing the conventional doubly stochasticity assumption, and only require the knowledge of the number of clients at initialization. We derive optimal instance-dependent regret upper bounds of order $\log{T}$ in both sub-gaussian and sub-exponential environments, and a nearly optimal mean-gap independent regret upper bound of order $\sqrt{T}\log T$ up to a $\log T$ factor. Importantly, our regret bounds hold with high probability and capture graph randomness, whereas prior works consider expected regret under assumptions and require more stringent reward distributions.