What do you Mean? The Role of the Mean Function in Bayesian Optimisation

TL;DR

This paper investigates how the choice of mean function in Gaussian process models affects the convergence of Bayesian optimisation, finding that simple worst-case mean functions often perform best in synthetic problems.

Contribution

It provides an empirical analysis of 8 different mean functions in Bayesian optimisation, highlighting the impact on convergence across synthetic and real-world problems.

Findings

Constant mean equal to the worst observed value improves convergence in synthetic problems.

More complex mean functions may be beneficial for real-world tasks.

Choice of mean function significantly influences optimisation performance.

Abstract

Bayesian optimisation is a popular approach for optimising expensive black-box functions. The next location to be evaluated is selected via maximising an acquisition function that balances exploitation and exploration. Gaussian processes, the surrogate models of choice in Bayesian optimisation, are often used with a constant prior mean function equal to the arithmetic mean of the observed function values. We show that the rate of convergence can depend sensitively on the choice of mean function. We empirically investigate 8 mean functions (constant functions equal to the arithmetic mean, minimum, median and maximum of the observed function evaluations, linear, quadratic polynomials, random forests and RBF networks), using 10 synthetic test problems and two real-world problems, and using the Expected Improvement and Upper Confidence Bound acquisition functions. We find that for design dimensions $\ge5$ using a constant mean function equal to the worst observed quality value is consistently the best choice on the synthetic problems considered. We argue that this worst-observed-quality function promotes exploitation leading to more rapid convergence. However, for the real-world tasks the more complex mean functions capable of modelling the fitness landscape may be effective, although there is no clearly optimum choice.