Value Directed Exploration in Multi-Armed Bandits with Structured Priors

TL;DR

This paper introduces a Bayesian multi-armed bandit algorithm that uses value-function-driven planning with structured priors, achieving near-optimal finite-time performance and demonstrating strong results in simulations.

Contribution

It presents a novel algorithm combining value functions and lookahead for structured priors, with theoretical guarantees and practical effectiveness.

Findings

Sub-linear performance guarantee established

Simulation results confirm effectiveness in structured prior problems

Approach is simple and adaptable to complex bandit scenarios

Abstract

Multi-armed bandits are a quintessential machine learning problem requiring the balancing of exploration and exploitation. While there has been progress in developing algorithms with strong theoretical guarantees, there has been less focus on practical near-optimal finite-time performance. In this paper, we propose an algorithm for Bayesian multi-armed bandits that utilizes value-function-driven online planning techniques. Building on previous work on UCB and Gittins index, we introduce linearly-separable value functions that take both the expected return and the benefit of exploration into consideration to perform n-step lookahead. The algorithm enjoys a sub-linear performance guarantee and we present simulation results that confirm its strength in problems with structured priors. The simplicity and generality of our approach makes it a strong candidate for analyzing more complex multi-armed bandit problems.