Multi-fidelity Bayesian Optimisation with Continuous Approximations

TL;DR

This paper introduces BOCA, a Bayesian optimisation method that efficiently leverages a continuous spectrum of approximations to accelerate black-box function optimization, outperforming existing methods in synthetic and real-world tasks.

Contribution

It develops a novel Bayesian optimisation approach for continuous approximations, extending multi-fidelity methods beyond finite sets and providing theoretical guarantees.

Findings

BOCA achieves lower regret than strategies ignoring approximations.

BOCA outperforms baseline methods in synthetic experiments.

BOCA demonstrates superior performance in real-world hyper-parameter tuning tasks.

Abstract

Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, \emph{multi-fidelity} methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. In many practical applications however, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data $N$ and/or few training iterations $T$. Here, the approximations are best viewed as arising out of a continuous two dimensional space $(N,T)$. In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.