Enhancing Model Based Derivative Free Optimization using Direct Search

TL;DR

This paper introduces a switching framework that combines Direct Search and model-based optimization methods to improve simulation-based optimization, with proven convergence and strong empirical results across ML and classical benchmarks.

Contribution

It proposes a novel switching approach that enhances model-based optimizers with Direct Search techniques, including convergence analysis and practical applications in machine learning and benchmark problems.

Findings

Proven asymptotic convergence of the switching method.

Strong numerical performance on ML tasks and classical test problems.

Effective warm-starting mechanism accelerates training in ML applications.

Abstract

We consider single and multiobjective simulation-based optimization problems. Simulation-based optimization has traditionally used both model-based and search-based methods, often in isolation. Model-based methods include trust region approaches and Bayesian optimization, while search methods include genetic algorithms and Direct Search-type techniques. In this work, we propose a switching framework that leverages Direct Search methods to enhance the performance of any model-based optimizer. Our contributions are twofold. First, in the single-objective setting, we analyze and prove the asymptotic convergence of the proposed switching approach. Second, motivated by applications in machine learning, we consider both classification and regression problems, where the objectives span accuracy, computational time, algorithmic bias, and sparsity. The models range from complex neural networks and decision trees to simpler KNN-type baselines. For machine learning in particular, we also introduce a warm-starting mechanism using weights from previous hyperparameter or architectural configurations to exploit problem structure and accelerate training. Finally, beyond ML tasks, we evaluate the method on the standard CUTEr test problems and compare its performance against classical Bayesian and trust region solvers. We observe consistently strong numerical performance, suggesting promise for the proposed switching-based approach.