Decentralized Cooperative Stochastic Bandits

TL;DR

This paper introduces a decentralized algorithm for multi-agent stochastic bandit problems that achieves near-optimal regret bounds by combining consensus procedures with UCB, improving over prior methods in simplicity and empirical performance.

Contribution

The paper presents a new decentralized algorithm that simplifies analysis, improves regret bounds, and requires less network information compared to previous approaches.

Findings

Regret bounded by centralized optimal plus spectral gap term

Algorithm outperforms prior work empirically

Achieves better regret bounds with simpler analysis

Abstract

We study a decentralized cooperative stochastic multi-armed bandit problem with $K$ arms on a network of $N$ agents. In our model, the reward distribution of each arm is the same for each agent and rewards are drawn independently across agents and time steps. In each round, each agent chooses an arm to play and subsequently sends a message to her neighbors. The goal is to minimize the overall regret of the entire network. We design a fully decentralized algorithm that uses an accelerated consensus procedure to compute (delayed) estimates of the average of rewards obtained by all the agents for each arm, and then uses an upper confidence bound (UCB) algorithm that accounts for the delay and error of the estimates. We analyze the regret of our algorithm and also provide a lower bound. The regret is bounded by the optimal centralized regret plus a natural and simple term depending on the spectral gap of the communication matrix. Our algorithm is simpler to analyze than those proposed in prior work and it achieves better regret bounds, while requiring less information about the underlying network. It also performs better empirically.