Information Directed Sampling for Stochastic Bandits with Graph Feedback

TL;DR

This paper introduces new graph-aware policies for stochastic multi-armed bandits with time-varying graph feedback, providing tighter theoretical bounds and demonstrating superior empirical performance over existing algorithms.

Contribution

It presents novel analysis of Thompson sampling and proposes Information Directed Sampling policies that adapt to graph structures, with new regret bounds for both deterministic and random graph models.

Findings

Proposed policies outperform existing algorithms in numerical tests.

Regret bounds scale with clique cover number and observation ratio.

First analytical results for bandits with random graph feedback.

Abstract

We consider stochastic multi-armed bandit problems with graph feedback, where the decision maker is allowed to observe the neighboring actions of the chosen action. We allow the graph structure to vary with time and consider both deterministic and Erd\H{o}s-R\'enyi random graph models. For such a graph feedback model, we first present a novel analysis of Thompson sampling that leads to tighter performance bound than existing work. Next, we propose new Information Directed Sampling based policies that are graph-aware in their decision making. Under the deterministic graph case, we establish a Bayesian regret bound for the proposed policies that scales with the clique cover number of the graph instead of the number of actions. Under the random graph case, we provide a Bayesian regret bound for the proposed policies that scales with the ratio of the number of actions over the expected number of observations per iteration. To the best of our knowledge, this is the first analytical result for stochastic bandits with random graph feedback. Finally, using numerical evaluations, we demonstrate that our proposed IDS policies outperform existing approaches, including adaptions of upper confidence bound, $\epsilon$-greedy and Exp3 algorithms.