Scalable neural network-based blackbox optimization

TL;DR

SNBO introduces a scalable neural network-based blackbox optimization method that avoids complex uncertainty estimation, adaptively balances exploration and exploitation, and outperforms existing algorithms in high-dimensional problems.

Contribution

The paper presents SNBO, a novel scalable NN-based blackbox optimization approach that eliminates the need for uncertainty estimation and improves efficiency in high-dimensional spaces.

Findings

SNBO achieves better function values than baselines in most tests.

SNBO requires 40-60% fewer function evaluations.

SNBO reduces runtime by at least an order of magnitude.

Abstract

Bayesian Optimization (BO) is a widely used approach for blackbox optimization that leverages a Gaussian process (GP) model and an acquisition function to guide future sampling. While effective in low-dimensional settings, BO faces scalability challenges in high-dimensional spaces and with large number of function evaluations due to the computational complexity of GP models. In contrast, neural networks (NNs) offer better scalability and can model complex functions, which led to the development of NN-based BO approaches. However, these methods typically rely on estimating model uncertainty in NN prediction -- a process that is often computationally intensive and complex, particularly in high dimensions. To address these limitations, a novel method, called scalable neural network-based blackbox optimization (SNBO), is proposed that does not rely on model uncertainty estimation. Specifically, SNBO adds new samples using separate criteria for exploration and exploitation, while adaptively controlling the sampling region to ensure efficient optimization. SNBO is evaluated on a range of optimization problems spanning from 10 to 102 dimensions and compared against four state-of-the-art baseline algorithms. Across the majority of test problems, SNBO attains function values better than the best-performing baseline algorithm, while requiring 40-60% fewer function evaluations and reducing the runtime by at least an order of magnitude.