Scalable Acceleration for Classification-Based Derivative-Free Optimization

TL;DR

This paper introduces RACE-CARS, a scalable and accelerated classification-based derivative-free optimization algorithm that improves efficiency through region shrinking, validated by experiments on synthetic and real-world tasks.

Contribution

It proposes RACE-CARS, an innovative algorithm that accelerates classification-based derivative-free optimization using region shrinking, supported by theoretical analysis and empirical validation.

Findings

RACE-CARS outperforms previous methods in efficiency.

Region shrinking significantly accelerates convergence.

Hyperparameter tuning guides improve practical application.

Abstract

Derivative-free optimization algorithms play an important role in scientific and engineering design optimization problems, especially when derivative information is not accessible. In this paper, we study the framework of sequential classification-based derivative-free optimization algorithms. By introducing learning theoretic concept hypothesis-target shattering rate, we revisit the computational complexity upper bound of SRACOS (Hu, Qian, and Yu 2017). Inspired by the revisited upper bound, we propose an algorithm named RACE-CARS, which adds a random region-shrinking step compared with SRACOS. We further establish theorems showing the acceleration by region shrinking. Experiments on the synthetic functions as well as black-box tuning for language-model-as-a-service demonstrate empirically the efficiency of RACE-CARS. An ablation experiment on the introduced hyperparameters is also conducted, revealing the mechanism of RACE-CARS and putting forward an empirical hyper-parameter tuning guidance.