Black-box optimization on hyper-rectangle using Recursive Modified Pattern Search and application to ROC-based Classification Problem

TL;DR

This paper introduces a new derivative-free optimization method based on pattern search for solving high-dimensional, multi-modal, and non-smooth problems with bounded parameters, demonstrating superior speed and effectiveness.

Contribution

The paper presents a novel recursive modified pattern search algorithm tailored for black-box optimization on hyper-rectangles, especially effective for high-dimensional, multi-modal functions.

Findings

Up to 40 times faster than genetic algorithms

Up to 368 times faster than simulated annealing

Successfully applied to optimize biomarkers for Alzheimer's disease

Abstract

In statistics, it is common to encounter multi-modal and non-smooth likelihood (or objective function) maximization problems, where the parameters have known upper and lower bounds. This paper proposes a novel derivative-free global optimization technique that can be used to solve those problems even when the objective function is not known explicitly or its derivatives are difficult or expensive to obtain. The technique is based on the pattern search algorithm, which has been shown to be effective for black-box optimization problems. The proposed algorithm works by iteratively generating new solutions from the current solution. The new solutions are generated by making movements along the coordinate axes of the constrained sample space. Before making a jump from the current solution to a new solution, the objective function is evaluated at several neighborhood points around the current solution. The best solution point is then chosen based on the objective function values at those points. Parallel threading can be used to make the algorithm more scalable. The performance of the proposed method is evaluated by optimizing up to 5000-dimensional multi-modal benchmark functions. The proposed algorithm is shown to be up to 40 and 368 times faster than genetic algorithm (GA) and simulated annealing (SA), respectively. The proposed method is also used to estimate the optimal biomarker combination from Alzheimer's disease data by maximizing the empirical estimates of the area under the receiver operating characteristic curve (AUC), outperforming the contextual popular alternative, known as step-down algorithm.