Derivative-Free Global Optimization Algorithms: Bayesian Method and Lipschitzian Approaches

TL;DR

This paper introduces derivative-free global optimization algorithms, focusing on Bayesian and Lipschitzian methods, as alternatives to gradient-based training for deep learning models, especially in non-convex scenarios.

Contribution

It provides an overview of Bayesian and Lipschitzian derivative-free optimization algorithms applicable to deep learning, highlighting their potential to overcome local optima issues.

Findings

Lipschitzian approaches like Shubert-Piyavskii, Direct, LIPO, and MCS are discussed.

Bayesian methods are introduced as promising global optimization techniques.

Remaining population-based algorithms will be detailed in a follow-up paper.

Abstract

In this paper, we will provide an introduction to the derivative-free optimization algorithms which can be potentially applied to train deep learning models. Existing deep learning model training is mostly based on the back propagation algorithm, which updates the model variables layers by layers with the gradient descent algorithm or its variants. However, the objective functions of deep learning models to be optimized are usually non-convex and the gradient descent algorithms based on the first-order derivative can get stuck into the local optima very easily. To resolve such a problem, various local or global optimization algorithms have been proposed, which can help improve the training of deep learning models greatly. The representative examples include the Bayesian methods, Shubert-Piyavskii algorithm, Direct, LIPO, MCS, GA, SCE, DE, PSO, ES, CMA-ES, hill climbing and simulated annealing, etc. One part of these algorithms will be introduced in this paper (including the Bayesian method and Lipschitzian approaches, e.g., Shubert-Piyavskii algorithm, Direct, LIPO and MCS), and the remaining algorithms (including the population based optimization algorithms, e.g., GA, SCE, DE, PSO, ES and CMA-ES, and random search algorithms, e.g., hill climbing and simulated annealing) will be introduced in the follow-up paper [18] in detail.