Parallelizing Contextual Bandits

TL;DR

This paper introduces parallel contextual bandit algorithms that enable faster exploration by proposing multiple decisions simultaneously, maintaining near-optimal regret while accelerating decision-making in practical applications.

Contribution

It develops a family of parallel bandit algorithms with regret bounds comparable to sequential methods, including new linear bandit algorithms that incorporate diversity in actions.

Findings

Algorithms achieve near-sequential regret with parallel proposals

Empirical results demonstrate effectiveness in materials discovery and biological design

Diversity-enhanced linear bandit algorithms improve exploration efficiency

Abstract

Standard approaches to decision-making under uncertainty focus on sequential exploration of the space of decisions. However, \textit{simultaneously} proposing a batch of decisions, which leverages available resources for parallel experimentation, has the potential to rapidly accelerate exploration. We present a family of (parallel) contextual bandit algorithms applicable to problems with bounded eluder dimension whose regret is nearly identical to their perfectly sequential counterparts -- given access to the same total number of oracle queries -- up to a lower-order ``burn-in" term. We further show these algorithms can be specialized to the class of linear reward functions where we introduce and analyze several new linear bandit algorithms which explicitly introduce diversity into their action selection. Finally, we also present an empirical evaluation of these parallel algorithms in several domains, including materials discovery and biological sequence design problems, to demonstrate the utility of parallelized bandits in practical settings.