Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

TL;DR

This paper introduces LADDER, a new Bayesian optimization method for combinatorial spaces that combines latent space representations with structure-aware kernels, leading to improved optimization performance.

Contribution

LADDER integrates structural information into the surrogate model via a novel structure-coupled kernel, enhancing BO over latent spaces for combinatorial optimization.

Findings

LADDER outperforms existing BO over latent space methods.

LADDER matches or exceeds state-of-the-art performance on real-world benchmarks.

The structure-coupled kernel improves surrogate modeling accuracy.

Abstract

We consider the problem of optimizing combinatorial spaces (e.g., sequences, trees, and graphs) using expensive black-box function evaluations. For example, optimizing molecules for drug design using physical lab experiments. Bayesian optimization (BO) is an efficient framework for solving such problems by intelligently selecting the inputs with high utility guided by a learned surrogate model. A recent BO approach for combinatorial spaces is through a reduction to BO over continuous spaces by learning a latent representation of structures using deep generative models (DGMs). The selected input from the continuous space is decoded into a discrete structure for performing function evaluation. However, the surrogate model over the latent space only uses the information learned by the DGM, which may not have the desired inductive bias to approximate the target black-box function. To overcome this drawback, this paper proposes a principled approach referred as LADDER. The key idea is to define a novel structure-coupled kernel that explicitly integrates the structural information from decoded structures with the learned latent space representation for better surrogate modeling. Our experiments on real-world benchmarks show that LADDER significantly improves over the BO over latent space method, and performs better or similar to state-of-the-art methods.