Unimodal Thompson Sampling for Graph-Structured Arms

TL;DR

This paper introduces a Bayesian Thompson Sampling algorithm for unimodal multi-armed bandit problems with graph structures, demonstrating superior performance over frequentist methods through theoretical analysis and experiments.

Contribution

It presents the first Bayesian algorithm for unimodal graph-structured MABs, achieving asymptotic optimality and outperforming existing frequentist approaches.

Findings

Thompson Sampling matches the lower regret bound.

Bayesian algorithms outperform frequentist methods in this setting.

Experimental results show improved performance across various graph structures.

Abstract

We study, to the best of our knowledge, the first Bayesian algorithm for unimodal Multi-Armed Bandit (MAB) problems with graph structure. In this setting, each arm corresponds to a node of a graph and each edge provides a relationship, unknown to the learner, between two nodes in terms of expected reward. Furthermore, for any node of the graph there is a path leading to the unique node providing the maximum expected reward, along which the expected reward is monotonically increasing. Previous results on this setting describe the behavior of frequentist MAB algorithms. In our paper, we design a Thompson Sampling-based algorithm whose asymptotic pseudo-regret matches the lower bound for the considered setting. We show that -as it happens in a wide number of scenarios- Bayesian MAB algorithms dramatically outperform frequentist ones. In particular, we provide a thorough experimental evaluation of the performance of our and state-of-the-art algorithms as the properties of the graph vary.