A General Stochastic Algorithmic Framework for Minimizing Expensive Black Box Objective Functions Based on Surrogate Models and Sensitivity Analysis

TL;DR

This paper introduces a new stochastic optimization framework that enhances the MSRS algorithm for expensive black-box functions, improving performance in high-dimensional problems through adaptive candidate point generation and sensitivity analysis.

Contribution

It extends the MSRS algorithm with a new candidate point generation method based on a novel distance measure and introduces SO-SA, an adaptive version that perturbs sensitive coordinates for better optimization.

Findings

The new algorithm outperforms existing methods on synthetic and real problems.

The convergence of the proposed method is theoretically proven.

Sensitivity-based perturbation improves optimization efficiency.

Abstract

We are focusing on bound constrained global optimization problems, whose objective functions are computationally expensive black-box functions and have multiple local minima. The recently popular Metric Stochastic Response Surface (MSRS) algorithm proposed by \cite{Regis2007SRBF} based on adaptive or sequential learning based on response surfaces is revisited and further extended for better performance in case of higher dimensional problems. Specifically, we propose a new way to generate the candidate points which the next function evaluation point is picked from according to the metric criteria, based on a new definition of distance, and prove the global convergence of the corresponding. Correspondingly, a more adaptive implementation of MSRS, named "SO-SA", is presented. "SO-SA" is is more likely to perturb those most sensitive coordinates when generating the candidate points, instead of perturbing all coordinates simultaneously. Numerical experiments on both synthetic problems and real problems demonstrate the advantages of our new algorithm, compared with many state of the art alternatives.}