The case for fully Bayesian optimisation in small-sample trials

TL;DR

Fully Bayesian optimisation offers a more robust and practical alternative to ML-II in small-sample trials, preventing failures and improving efficiency in expensive black-box function evaluations.

Contribution

The paper advocates for fully Bayesian optimisation over ML-II, demonstrating its robustness, simplicity, and practicality through experiments, especially in small-sample settings.

Findings

FBO is more robust than ML-II in small-sample trials.

Failures of ML-II are more common than previously thought.

FBO is easy to implement and computationally feasible.

Abstract

While sample efficiency is the main motive for use of Bayesian optimisation when black-box functions are expensive to evaluate, the standard approach based on type II maximum likelihood (ML-II) may fail and result in disappointing performance in small-sample trials. The paper provides three compelling reasons to adopt fully Bayesian optimisation (FBO) as an alternative. First, failures of ML-II are more commonplace than implied by the existing studies using the contrived settings. Second, FBO is more robust than ML-II, and the price of robustness is almost trivial. Third, FBO has become simple to implement and fast enough to be practical. The paper supports the argument using relevant experiments, which reflect the current practice regarding models, algorithms, and software platforms. Since the benefits seem to outweigh the costs, researchers should consider adopting FBO for their applications so that they can guard against potential failures that end up wasting precious research resources.