Long-run Behaviour of Multi-fidelity Bayesian Optimisation

TL;DR

This paper investigates the long-term behavior of multi-fidelity Bayesian optimisation, revealing potential under-performance issues in certain scenarios through empirical analysis and benchmark studies.

Contribution

It provides the first systematic investigation into the long-run performance of MFBO, highlighting scenarios where it may underperform compared to SFBO.

Findings

MFBO can underperform in certain long-term scenarios

Empirical results identify conditions leading to under-performance

Benchmark study illustrates scenarios and reasons for under-performance

Abstract

Multi-fidelity Bayesian Optimisation (MFBO) has been shown to generally converge faster than single-fidelity Bayesian Optimisation (SFBO) (Poloczek et al. (2017)). Inspired by recent benchmark papers, we are investigating the long-run behaviour of MFBO, based on observations in the literature that it might under-perform in certain scenarios (Mikkola et al. (2023), Eggensperger et al. (2021)). An under-performance of MBFO in the long-run could significantly undermine its application to many research tasks, especially when we are not able to identify when the under-performance begins. We create a simple benchmark study, showcase empirical results and discuss scenarios and possible reasons of under-performance.