Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization

TL;DR

This paper introduces a unifying framework for combinatorial kernels in Bayesian Optimization based on heat kernels, demonstrating their effectiveness and invariance properties, leading to state-of-the-art results.

Contribution

The authors develop a systematic derivation of heat kernels for combinatorial domains and show many existing kernels are related or equivalent to them.

Findings

Heat kernels are related or equivalent to many successful combinatorial kernels.

Heat kernels are insensitive to the location of the optima, unlike some algorithms.

A simple pipeline using heat kernels achieves state-of-the-art results.

Abstract

Bayesian Optimization (BO) has the potential to solve various combinatorial tasks, ranging from materials science to neural architecture search. However, BO requires specialized kernels to effectively model combinatorial domains. Recent efforts have introduced several combinatorial kernels, but the relationships among them are not well understood. To bridge this gap, we develop a unifying framework based on heat kernels, which we derive in a systematic way and express as simple closed-form expressions. Using this framework, we prove that many successful combinatorial kernels are either related or equivalent to heat kernels, and validate this theoretical claim in our experiments. Moreover, our analysis confirms and extends the results presented in Bounce: certain algorithms' performance decreases substantially when the unknown optima of the function do not have a certain structure. In contrast, heat kernels are not sensitive to the location of the optima. Lastly, we show that a fast and simple pipeline, relying on heat kernels, is able to achieve state-of-the-art results, matching or even outperforming certain slow or complex algorithms.