Correlated bandits or: How to minimize mean-squared error online

TL;DR

This paper introduces the correlated bandit problem focused on minimizing mean-squared error in estimating all arms, emphasizing correlation structure learning over traditional mean maximization, with theoretical bounds and an algorithmic solution.

Contribution

It formulates the correlated bandit problem, develops an MSE estimator, and proposes an algorithm with theoretical error bounds, advancing beyond traditional bandit objectives.

Findings

The MSE estimator concentrates exponentially around the true MSE.

The proposed algorithm effectively identifies the best arm with bounded error probability.

Fundamental performance limits for the correlated bandit problem are derived.

Abstract

While the objective in traditional multi-armed bandit problems is to find the arm with the highest mean, in many settings, finding an arm that best captures information about other arms is of interest. This objective, however, requires learning the underlying correlation structure and not just the means of the arms. Sensors placement for industrial surveillance and cellular network monitoring are a few applications, where the underlying correlation structure plays an important role. Motivated by such applications, we formulate the correlated bandit problem, where the objective is to find the arm with the lowest mean-squared error (MSE) in estimating all the arms. To this end, we derive first an MSE estimator, based on sample variances and covariances, and show that our estimator exponentially concentrates around the true MSE. Under a best-arm identification framework, we propose a successive rejects type algorithm and provide bounds on the probability of error in identifying the best arm. Using minmax theory, we also derive fundamental performance limits for the correlated bandit problem.