Enhancing Trust-Region Bayesian Optimization via Newton Methods

TL;DR

This paper introduces a novel high-dimensional Bayesian Optimization method that combines trust-region strategies with Newton methods, utilizing gradients and Hessians from a global Gaussian Process to improve sampling efficiency and convergence.

Contribution

It proposes constructing multiple local quadratic models using global GP derivatives and solving quadratic programs for sample selection, addressing high-dimensional challenges.

Findings

Enhanced optimization efficiency over existing high-dimensional BO methods

Outperforms state-of-the-art techniques on synthetic and real-world problems

Provides convergence guarantees for the proposed approach

Abstract

Bayesian Optimization (BO) has been widely applied to optimize expensive black-box functions while retaining sample efficiency. However, scaling BO to high-dimensional spaces remains challenging. Existing literature proposes performing standard BO in multiple local trust regions (TuRBO) for heterogeneous modeling of the objective function and avoiding over-exploration. Despite its advantages, using local Gaussian Processes (GPs) reduces sampling efficiency compared to a global GP. To enhance sampling efficiency while preserving heterogeneous modeling, we propose to construct multiple local quadratic models using gradients and Hessians from a global GP, and select new sample points by solving the bound-constrained quadratic program. Additionally, we address the issue of vanishing gradients of GPs in high-dimensional spaces. We provide a convergence analysis and demonstrate through experimental results that our method enhances the efficacy of TuRBO and outperforms a wide range of high-dimensional BO techniques on synthetic functions and real-world applications.