High-Dimensional Bayesian Optimization via Random Projection of Manifold Subspaces

TL;DR

This paper introduces a novel high-dimensional Bayesian Optimization method that combines random projections with geometry-aware semi-supervised learning to efficiently optimize functions on low-dimensional manifolds embedded in high-dimensional spaces.

Contribution

It proposes a new approach that integrates random linear projections with nonlinear manifold representation learning, improving BO performance in high dimensions.

Findings

Outperforms existing high-dimensional BO baselines on synthetic functions.

Effectively exploits manifold geometry for better representation learning.

Reduces computational complexity in high-dimensional optimization tasks.

Abstract

Bayesian Optimization (BO) is a popular approach to optimizing expensive-to-evaluate black-box functions. Despite the success of BO, its performance may decrease exponentially as the dimensionality increases. A common framework to tackle this problem is to assume that the objective function depends on a limited set of features that lie on a low-dimensional manifold embedded in the high-dimensional ambient space. The latent space can be linear or more generally nonlinear. To learn feature mapping, existing works usually use an encode-decoder framework which is either computationally expensive or susceptible to overfittting when the labeled data is limited. This paper proposes a new approach for BO in high dimensions by exploiting a new representation of the objective function. Our approach combines a random linear projection to reduce the dimensionality, with a representation learning of the nonlinear manifold. When the geometry of the latent manifold is available, a solution to exploit this geometry is proposed for representation learning. In contrast, we use a neural network. To mitigate overfitting by using the neural network, we train the feature mapping in a geometry-aware semi-supervised manner. Our approach enables efficient optimizing of BO's acquisition function in the low-dimensional space, with the advantage of projecting back to the original high-dimensional space compared to existing works in the same setting. Finally, we show empirically that our algorithm outperforms other high-dimensional BO baselines in various synthetic functions and real applications.