Cooperative Bandit Learning in Directed Networks with Arm-Access Constraints

TL;DR

This paper introduces a distributed UCB algorithm for multi-agent bandit problems with heterogeneous arm access and directed communication networks, achieving logarithmic regret while accounting for network and accessibility constraints.

Contribution

It presents a novel consensus-based UCB method that handles partial arm access and asymmetric communication, with theoretical regret guarantees.

Findings

Logarithmic regret bounds are established for each agent.

The algorithm's performance depends on network mixing properties.

Heterogeneous arm access influences cooperative learning efficiency.

Abstract

Sequential decision-making under uncertainty often involves multiple agents learning which actions (arms) yield the highest rewards through repeated interaction with a stochastic environment. This setting is commonly modeled by cooperative multi-agent multi-armed bandit problems, where agents explore and share information without centralized coordination. In many realistic systems, agents have heterogeneous capabilities that limit their access to subsets of arms and communicate over asymmetric networks represented by directed graphs. In this work, we study multi-agent multi-armed bandit problems with partial arm access, where agents explore and exploit only the arms available to them while exchanging information with neighbors. We propose a distributed consensus-based upper confidence bound (UCB) algorithm that accounts for both the arm accessibility structure and network asymmetry. Our approach employs a mass-preserving information mixing mechanism, ensuring that reward estimates remain unbiased across the network despite accessibility constraints and asymmetric information flow. Under standard stochastic assumptions, we establish logarithmic regret for every agent, with explicit dependence on network mixing properties and arm accessibility constraints. These results quantify how heterogeneous arm access and directed communication shape cooperative learning performance.