Empirical Bayesian Multi-Bandit Learning

TL;DR

This paper introduces a hierarchical Bayesian framework for multi-task bandit learning, estimating covariance structures to improve decision-making and achieve lower regret in complex environments.

Contribution

It proposes an empirical Bayesian approach to learn covariance matrices across bandits, along with two algorithms, ebmTS and ebmUCB, with regret bounds and superior performance.

Findings

Algorithms outperform existing methods in synthetic datasets.

Lower cumulative regret achieved in real-world experiments.

Effective balance of exploration and exploitation demonstrated.

Abstract

Multi-task learning in contextual bandits has attracted significant research interest due to its potential to enhance decision-making across multiple related tasks by leveraging shared structures and task-specific heterogeneity. In this article, we propose a novel hierarchical Bayesian framework for learning in various bandit instances. This framework captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations. Unlike previous methods that overlook the learning of the covariance structure across bandits, we introduce an empirical Bayesian approach to estimate the covariance matrix of the prior distribution. This enhances both the practicality and flexibility of learning across multi-bandits. Building on this approach, we develop two efficient algorithms: ebmTS (Empirical Bayesian Multi-Bandit Thompson Sampling) and ebmUCB (Empirical Bayesian Multi-Bandit Upper Confidence Bound), both of which incorporate the estimated prior into the decision-making process. We provide the frequentist regret upper bounds for the proposed algorithms, thereby filling a research gap in the field of multi-bandit problems. Extensive experiments on both synthetic and real-world datasets demonstrate the superior performance of our algorithms, particularly in complex environments. Our methods achieve lower cumulative regret compared to existing techniques, highlighting their effectiveness in balancing exploration and exploitation across multi-bandits.