Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization

TL;DR

This paper presents a modular framework for Mixed-variable and Combinatorial Bayesian Optimization, enabling systematic benchmarking and evaluation of diverse MCBO algorithms through extensive experiments on synthetic and real-world tasks.

Contribution

The authors introduce a flexible, easy-to-use MCBO framework that facilitates the creation and benchmarking of numerous MCBO algorithms, addressing previous evaluation limitations.

Findings

A superior MCBO primitive combination outperforms existing methods.

Model fit and trust region significantly impact optimization performance.

Over 4000 experiments validate the framework's effectiveness.

Abstract

This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at \url{https://github.com/huawei-noah/HEBO/tree/master/MCBO}.