BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings

TL;DR

BOIDS is a new high-dimensional Bayesian Optimization method that uses direction lines and subspace embeddings to improve efficiency and scalability, outperforming existing methods on benchmarks.

Contribution

Introduces BOIDS, a novel high-dimensional BO algorithm combining line-based optimization, adaptive line selection, and subspace embeddings with theoretical convergence analysis.

Findings

BOIDS outperforms state-of-the-art baselines on synthetic benchmarks.

The method effectively scales to high-dimensional problems.

Theoretical analysis confirms convergence properties.

Abstract

When it comes to expensive black-box optimization problems, Bayesian Optimization (BO) is a well-known and powerful solution. Many real-world applications involve a large number of dimensions, hence scaling BO to high dimension is of much interest. However, state-of-the-art high-dimensional BO methods still suffer from the curse of dimensionality, highlighting the need for further improvements. In this work, we introduce BOIDS, a novel high-dimensional BO algorithm that guides optimization by a sequence of one-dimensional direction lines using a novel tailored line-based optimization procedure. To improve the efficiency, we also propose an adaptive selection technique to identify most optimal lines for each round of line-based optimization. Additionally, we incorporate a subspace embedding technique for better scaling to high-dimensional spaces. We further provide theoretical analysis of our proposed method to analyze its convergence property. Our extensive experimental results show that BOIDS outperforms state-of-the-art baselines on various synthetic and real-world benchmark problems.