Improving Thompson Sampling via Information Relaxation for Budgeted Multi-armed Bandits

TL;DR

This paper enhances Bayesian budgeted multi-armed bandit algorithms by integrating information relaxation techniques into Thompson Sampling, leading to more resource-aware decision-making and improved performance benchmarks.

Contribution

It introduces a novel framework that generalizes Thompson Sampling using information relaxation, creating algorithms that better optimize resource constraints in budgeted bandit problems.

Findings

Algorithms outperform traditional BTS in simulations.

Improved benchmarks provide better performance measures.

Effective in real-world resource allocation scenarios.

Abstract

We consider a Bayesian budgeted multi-armed bandit problem, in which each arm consumes a different amount of resources when selected and there is a budget constraint on the total amount of resources that can be used. Budgeted Thompson Sampling (BTS) offers a very effective heuristic to this problem, but its arm-selection rule does not take into account the remaining budget information. We adopt \textit{Information Relaxation Sampling} framework that generalizes Thompson Sampling for classical $K$-armed bandit problems, and propose a series of algorithms that are randomized like BTS but more carefully optimize their decisions with respect to the budget constraint. In a one-to-one correspondence with these algorithms, a series of performance benchmarks that improve the conventional benchmark are also suggested. Our theoretical analysis and simulation results show that our algorithms (and our benchmarks) make incremental improvements over BTS (respectively, the conventional benchmark) across various settings including a real-world example.