Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?

TL;DR

This paper explores data-independent, random decomposition sampling in high-dimensional Bayesian optimization, demonstrating theoretical advantages and empirical improvements over data-driven methods and existing algorithms.

Contribution

It introduces a random decomposition sampler with theoretical guarantees and develops RDUCB, a simple yet effective algorithm that outperforms state-of-the-art methods.

Findings

Random decomposition sampling offers favorable theoretical guarantees.

RDUCB outperforms previous methods on benchmark tasks.

Integration with HEBO improves high-dimensional optimization results.

Abstract

Learning decompositions of expensive-to-evaluate black-box functions promises to scale Bayesian optimisation (BO) to high-dimensional problems. However, the success of these techniques depends on finding proper decompositions that accurately represent the black-box. While previous works learn those decompositions based on data, we investigate data-independent decomposition sampling rules in this paper. We find that data-driven learners of decompositions can be easily misled towards local decompositions that do not hold globally across the search space. Then, we formally show that a random tree-based decomposition sampler exhibits favourable theoretical guarantees that effectively trade off maximal information gain and functional mismatch between the actual black-box and its surrogate as provided by the decomposition. Those results motivate the development of the random decomposition upper-confidence bound algorithm (RDUCB) that is straightforward to implement - (almost) plug-and-play - and, surprisingly, yields significant empirical gains compared to the previous state-of-the-art on a comprehensive set of benchmarks. We also confirm the plug-and-play nature of our modelling component by integrating our method with HEBO, showing improved practical gains in the highest dimensional tasks from Bayesmark.