StepDIRECT -- A Derivative-Free Optimization Method for Stepwise Functions

TL;DR

The paper introduces StepDIRECT, a derivative-free optimization algorithm tailored for stepwise functions, combining local variability insights and stochastic local search to enhance convergence and solution quality.

Contribution

It extends the DIRECT algorithm with new criteria and stochastic search techniques, specifically designed for stepwise landscapes, and proves its global convergence.

Findings

Competitive performance on hyper-parameter tuning tasks

Effective in optimizing random forest models

Outperforms some baseline DFO methods

Abstract

In this paper, we propose the StepDIRECT algorithm for derivative-free optimization (DFO), in which the black-box objective function has a stepwise landscape. Our framework is based on the well-known DIRECT algorithm. By incorporating the local variability to explore the flatness, we provide a new criterion to select the potentially optimal hyper-rectangles. In addition, we introduce a stochastic local search algorithm performing on potentially optimal hyper-rectangles to improve the solution quality and convergence speed. Global convergence of the StepDIRECT algorithm is provided. Numerical experiments on optimization for random forest models and hyper-parameter tuning are presented to support the efficacy of our algorithm. The proposed StepDIRECT algorithm shows competitive performance results compared with other state-of-the-art baseline DFO methods including the original DIRECT algorithm.