Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization

TL;DR

SA-REMBO enhances high-dimensional Bayesian Optimization by adaptively selecting multiple local embeddings, effectively handling nonstationarity and complex structures in the objective landscape, outperforming traditional methods.

Contribution

Introduces a novel framework that generalizes REMBO with multiple embeddings and an index variable, capturing local structures and nonstationarity in high-dimensional BO.

Findings

Outperforms traditional REMBO and low-rank BO methods on benchmarks.

Effectively captures local structure and nonstationarity.

Theoretically analyzed kernel expressiveness and stability.

Abstract

Bayesian Optimization (BO) in high-dimensional spaces remains fundamentally limited by the curse of dimensionality and the rigidity of global low-dimensional assumptions. While Random EMbedding Bayesian Optimization (REMBO) mitigates this via linear projections into low-dimensional subspaces, it typically assumes a single global embedding and a stationary objective. In this work, we introduce Self-Adaptive embedding REMBO (SA-REMBO), a novel framework that generalizes REMBO to support multiple random Gaussian embeddings, each capturing a different local subspace structure of the high-dimensional objective. An index variable governs the embedding choice and is jointly modeled with the latent optimization variable via a product kernel in a Gaussian Process surrogate. This enables the optimizer to adaptively select embeddings conditioned on location, effectively capturing locally varying effective dimensionality, nonstationarity, and heteroscedasticity in the objective landscape. We theoretically analyze the expressiveness and stability of the index-conditioned product kernel and empirically demonstrate the advantage of our method across synthetic and real-world high-dimensional benchmarks, where traditional REMBO and other low-rank BO methods fail. Our results establish SA-REMBO as a powerful and flexible extension for scalable BO in complex, structured design spaces.