Multi-armed Bandit Learning on a Graph

TL;DR

This paper introduces the graph bandit, a new extension of the multi-armed bandit problem where an agent navigates a graph to maximize rewards, and presents a learning algorithm with near-optimal regret bounds.

Contribution

The paper proposes G-UCB, a novel algorithm for graph bandits that achieves tight regret bounds, extending MAB theory to graph-structured decision spaces.

Findings

G-UCB achieves near-optimal regret bounds of $O( ext{sqrt{|S|}T log T} + D|S| log T)$.

Theoretical analysis shows the regret bound matches the lower bound $ ext{Omega}( ext{sqrt{|S|}T})$ up to logarithmic factors.

Numerical experiments demonstrate G-UCB outperforms existing benchmarks.

Abstract

The multi-armed bandit(MAB) problem is a simple yet powerful framework that has been extensively studied in the context of decision-making under uncertainty. In many real-world applications, such as robotic applications, selecting an arm corresponds to a physical action that constrains the choices of the next available arms (actions). Motivated by this, we study an extension of MAB called the graph bandit, where an agent travels over a graph to maximize the reward collected from different nodes. The graph defines the agent's freedom in selecting the next available nodes at each step. We assume the graph structure is fully available, but the reward distributions are unknown. Built upon an offline graph-based planning algorithm and the principle of optimism, we design a learning algorithm, G-UCB, that balances long-term exploration-exploitation using the principle of optimism. We show that our proposed algorithm achieves $O(\sqrt{|S|T\log(T)}+D|S|\log T)$ learning regret, where $|S|$ is the number of nodes and $D$ is the diameter of the graph, which matches the theoretical lower bound $\Omega(\sqrt{|S|T})$ up to logarithmic factors. To our knowledge, this result is among the first tight regret bounds in non-episodic, un-discounted learning problems with known deterministic transitions. Numerical experiments confirm that our algorithm outperforms several benchmarks.