Preferential Bayesian Optimization

TL;DR

Preferential Bayesian Optimization (PBO) is a novel framework that efficiently finds optima of functions through pairwise user preferences, significantly reducing the number of comparisons needed compared to traditional methods.

Contribution

This paper introduces PBO, extending Bayesian optimization to scenarios where only preference data is available, and models preferences with Gaussian processes for improved efficiency.

Findings

PBO requires fewer comparisons to find the optimum.

Modeling correlations in PBO improves performance.

PBO generalizes previous dueling approaches.

Abstract

Bayesian optimization (BO) has emerged during the last few years as an effective approach to optimizing black-box functions where direct queries of the objective are expensive. In this paper we consider the case where direct access to the function is not possible, but information about user preferences is. Such scenarios arise in problems where human preferences are modeled, such as A/B tests or recommender systems. We present a new framework for this scenario that we call Preferential Bayesian Optimization (PBO) which allows us to find the optimum of a latent function that can only be queried through pairwise comparisons, the so-called duels. PBO extends the applicability of standard BO ideas and generalizes previous discrete dueling approaches by modeling the probability of the winner of each duel by means of a Gaussian process model with a Bernoulli likelihood. The latent preference function is used to define a family of acquisition functions that extend usual policies used in BO. We illustrate the benefits of PBO in a variety of experiments, showing that PBO needs drastically fewer comparisons for finding the optimum. According to our experiments, the way of modeling correlations in PBO is key in obtaining this advantage.