KSOS-BO: Improving Sampling in Bayesian Optimization via Kernel Sum of Squares

TL;DR

KSOS-BO introduces a kernel-based semidefinite programming approach to optimize Bayesian Optimization acquisition functions, significantly improving sample efficiency and convergence speed across diverse benchmark functions.

Contribution

It presents a novel structured global search method for acquisition optimization in Bayesian Optimization using kernel sum of squares and semidefinite programming.

Findings

Outperforms derivative-free baselines with an 81.16% average regret reduction.

Achieves a 93.55% average improvement in wall-clock time to reach high-quality solutions.

Excels on multimodal and ill-conditioned functions, demonstrating broad applicability.

Abstract

Bayesian Optimization (BO) is an effective framework for globally optimizing functions whose evaluations are expensive. It is particularly effective for optimizing functions defined over continuous domains and explicitly handles stochastic noise in evaluations. As a result, it is widely applied in areas such as hyperparameter tuning, robotics policy search, and scientific experiment design, where sample efficiency is essential. Its two-step procedure consists of model fitting followed by optimization of the acquisition function, which is often treated as a generic black-box problem despite its structured nature. In this work, we introduce KSOS-BO, a kernel-based derivative-free framework for BO acquisition optimization. KSOS-BO formulates the optimization of the acquisition function as a semidefinite program with kernel-induced representations, enabling a structured global search. Across a diverse set of benchmark functions with varying landscape properties, KSOS-BO consistently outperforms derivative-free baselines using Sobol Search, Differential Evolution, or CMA-ES to optimize the acquisition function, achieving an average regret improvement of 81.16% on 10/15 benchmarks. In particular, KSOS-BO demonstrates strong performance in highly multimodal and unimodal but ill-conditioned functions, indicating its applicability to diverse landscape structures. Despite a higher per-iteration computational cost, it converges faster in wall-clock time with an average improvement of 93.55% on 10/15 benchmarks, as it reaches high-quality solutions with fewer evaluations. Limitations include reduced effectiveness on functions with steep drops or plate-shaped regions.