A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes

TL;DR

This paper introduces MF-MI-Greedy, a novel multi-fidelity Bayesian optimization framework that models complex dependencies among multiple information sources using additive Gaussian processes, optimizing efficiently with theoretical guarantees.

Contribution

It develops a principled algorithm for multi-fidelity Bayesian optimization with complex dependencies, incorporating cost-sensitive mutual information and providing regret guarantees.

Findings

Achieves low regret in theoretical analysis.

Demonstrates strong empirical performance on synthetic datasets.

Effective in real-world optimization scenarios.

Abstract

How can we efficiently gather information to optimize an unknown function, when presented with multiple, mutually dependent information sources with different costs? For example, when optimizing a robotic system, intelligently trading off computer simulations and real robot testings can lead to significant savings. Existing methods, such as multi-fidelity GP-UCB or Entropy Search-based approaches, either make simplistic assumptions on the interaction among different fidelities or use simple heuristics that lack theoretical guarantees. In this paper, we study multi-fidelity Bayesian optimization with complex structural dependencies among multiple outputs, and propose MF-MI-Greedy, a principled algorithmic framework for addressing this problem. In particular, we model different fidelities using additive Gaussian processes based on shared latent structures with the target function. Then we use cost-sensitive mutual information gain for efficient Bayesian global optimization. We propose a simple notion of regret which incorporates the cost of different fidelities, and prove that MF-MI-Greedy achieves low regret. We demonstrate the strong empirical performance of our algorithm on both synthetic and real-world datasets.