Dimensionality Reduction Techniques for Global Bayesian Optimisation

TL;DR

This paper investigates the use of Variational Autoencoders for dimensionality reduction in Bayesian Optimization, demonstrating improved performance with structured latent spaces and integrating Sequential Domain Reduction in a GPU environment.

Contribution

It introduces a VAE-based LSBO framework with key corrections, broadens its application scope, and incorporates SDR into LSBO for the first time in a GPU setting.

Findings

Structured latent manifolds enhance BO performance

VAE-based LSBO outperforms linear projection methods

Sequential Domain Reduction improves optimization efficiency

Abstract

Bayesian Optimisation (BO) is a state-of-the-art global optimisation technique for black-box problems where derivative information is unavailable, and sample efficiency is crucial. However, improving the general scalability of BO has proved challenging. Here, we explore Latent Space Bayesian Optimisation (LSBO), that applies dimensionality reduction to perform BO in a reduced-dimensional subspace. While early LSBO methods used (linear) random projections (Wang et al., 2013), we employ Variational Autoencoders (VAEs) to manage more complex data structures and general DR tasks. Building on Grosnit et. al. (2021), we analyse the VAE-based LSBO framework, focusing on VAE retraining and deep metric loss. We suggest a few key corrections in their implementation, originally designed for tasks such as molecule generation, and reformulate the algorithm for broader optimisation purposes. Our numerical results show that structured latent manifolds improve BO performance. Additionally, we examine the use of the Mat\'{e}rn-$\frac{5}{2}$ kernel for Gaussian Processes in this LSBO context. We also integrate Sequential Domain Reduction (SDR), a standard global optimization efficiency strategy, into BO. SDR is included in a GPU-based environment using \textit{BoTorch}, both in the original and VAE-generated latent spaces, marking the first application of SDR within LSBO.