Output-Weighted Sampling for Multi-Armed Bandits with Extreme Payoffs

TL;DR

This paper introduces a novel output-weighted sampling method for multi-armed bandits that emphasizes exploration of extreme payoffs by modeling rewards with Gaussian processes and using a new UCB-based acquisition function.

Contribution

The paper proposes a new output-weighted sampling approach with a Gaussian process-based UCB for better exploration of extreme rewards in bandit problems.

Findings

Improved exploration of extreme payoffs in synthetic benchmarks.

Effective application to noisy sensor network data.

Provides a JAX library for efficient Gaussian process bandit optimization.

Abstract

We present a new type of acquisition functions for online decision making in multi-armed and contextual bandit problems with extreme payoffs. Specifically, we model the payoff function as a Gaussian process and formulate a novel type of upper confidence bound (UCB) acquisition function that guides exploration towards the bandits that are deemed most relevant according to the variability of the observed rewards. This is achieved by computing a tractable likelihood ratio that quantifies the importance of the output relative to the inputs and essentially acts as an \textit{attention mechanism} that promotes exploration of extreme rewards. We demonstrate the benefits of the proposed methodology across several synthetic benchmarks, as well as a realistic example involving noisy sensor network data. Finally, we provide a JAX library for efficient bandit optimization using Gaussian processes.